Today's Posts Follow Us On Twitter! TFL Members on Twitter  
Forum search: Advanced Search  
Navigation
Marketplace
  Members Login:
Lost password?
  Forum Statistics:
Forum Members: 24,256
Total Threads: 106,575
Total Posts: 592,698
There are 86 users currently browsing (tf).
 
  Our Partners:
 
  TalkFreelance     Design and Development     Programming :

which language do i need to use for processing 500gb of data?

Thread title: which language do i need to use for processing 500gb of data?
Reply  
Page 1 of 2 1 2 >
    Thread tools Search this thread Display Modes  
06-03-2012, 08:58 AM
#1
unikorndesigns is offline unikorndesigns
unikorndesigns's Avatar
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
 
Posts: 41
iTrader: 0 / 0%
 

unikorndesigns is on a distinguished road

  Old  which language do i need to use for processing 500gb of data?

i have a server filled with 500GB of docx, pptx, html files, pdfs etc. I wanted to create a search engine where all these files will b crawled nd indexed! I wud like to know which programming language should i rely on? Am an expert php developer! I found php works good, but it takes alot of time to crawl nd process each file! Any solution or suggestions? Do i need to code in python, perl, c, c++, java or in ruby on rails?

Reply With Quote
06-03-2012, 01:37 PM
#2
Village Genius is offline Village Genius
Village Genius's Avatar
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
 
Posts: 6,892
iTrader: 18 / 100%
 

Village Genius will become famous soon enough

  Old

Nothing is going to search half a terabyte quickly. Your algorithms are going to matter more than the language you do it in. C will be the fastest but if you just iterate though a directory it's all going to be slow. You're going to need to index these files and search through that.

Reply With Quote
06-03-2012, 02:57 PM
#3
unikorndesigns is offline unikorndesigns
unikorndesigns's Avatar
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
 
Posts: 41
iTrader: 0 / 0%
 

unikorndesigns is on a distinguished road

  Old

i will be creating a crawler to index these files, but i need a programming language that crawls and indexes these files realy fast. The searchng is being done using php which i dont mind changing if there is any other language whch can do better than php! Btw if i were to use C language, how would i execute the program on a server and how wil i be able to index these files on a MySQL database?

Reply With Quote
06-03-2012, 05:45 PM
#4
Village Genius is offline Village Genius
Village Genius's Avatar
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
 
Posts: 6,892
iTrader: 18 / 100%
 

Village Genius will become famous soon enough

  Old

PHP has command line access (shell_exec()) so you can run the programs you make, there are also MySQL libraries for C. If you aren't familiar with Linux programming doing it C will be no trivial task; everything is hard in C..

What you need to do is see which parts of your program are taking the longest and optimize them.

Reply With Quote
06-04-2012, 05:34 AM
#5
unikorndesigns is offline unikorndesigns
unikorndesigns's Avatar
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
 
Posts: 41
iTrader: 0 / 0%
 

unikorndesigns is on a distinguished road

  Old

i just found that c language is pretty fast and it would be great if u could show me an example of executing a c program using php! I tried googling it, but nothng seemed to work.

Reply With Quote
06-04-2012, 06:54 AM
#6
.Nick is offline .Nick
.Nick's Avatar
Status: Yawn
Join date: Feb 2005
Location:
Expertise: Drupal
Software: Chromium, Komodo Edit, Eclipse
 
Posts: 1,159
iTrader: 3 / 100%
 

.Nick is on a distinguished road

  Old

Originally Posted by unikorndesigns View Post
i just found that c language is pretty fast and it would be great if u could show me an example of executing a c program using php! I tried googling it, but nothng seemed to work.

http://us3.php.net/manual/en/function.shell-exec.php

PHP Code:
<?php
$output 
shell_exec('ls -la');
echo 
"<pre>$output</pre>";
?>
Also, see this thread:
http://talkfreelance.com/showthread.php?t=80592

06-04-2012, 08:01 AM
#7
unikorndesigns is offline unikorndesigns
unikorndesigns's Avatar
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
 
Posts: 41
iTrader: 0 / 0%
 

unikorndesigns is on a distinguished road

  Old

i dont see a C program inside it! If am nt wrong, that has got to be linux coding, ryt?

Reply With Quote
06-04-2012, 02:32 PM
#8
.Nick is offline .Nick
.Nick's Avatar
Status: Yawn
Join date: Feb 2005
Location:
Expertise: Drupal
Software: Chromium, Komodo Edit, Eclipse
 
Posts: 1,159
iTrader: 3 / 100%
 

.Nick is on a distinguished road

  Old

Originally Posted by unikorndesigns View Post
i dont see a C program inside it! If am nt wrong, that has got to be linux coding, ryt?
C programs are compiled into executables.

Reply With Quote
06-04-2012, 03:00 PM
#9
Village Genius is offline Village Genius
Village Genius's Avatar
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
 
Posts: 6,892
iTrader: 18 / 100%
 

Village Genius will become famous soon enough

  Old

Originally Posted by unikorndesigns View Post
i just found that c language is pretty fast and it would be great if u could show me an example of executing a c program using php! I tried googling it, but nothng seemed to work.
C needs to be compiled into a program. If you're in a linux setup you're going to need to learn how to use the GCC compiler. There's tons of resources on google. There's also IDEs out there but I usually just use Vim and GCC on the command line.

Reply With Quote
06-04-2012, 04:01 PM
#10
unikorndesigns is offline unikorndesigns
unikorndesigns's Avatar
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
 
Posts: 41
iTrader: 0 / 0%
 

unikorndesigns is on a distinguished road

  Old

oh ok! I am using Wampserver! Is there a way to do for windows setup! I had tried google but the results did not seem to work, i suppose its because of da windows setup! Any soln that wud work with windows?

Reply With Quote
Reply  
Page 1 of 2 1 2 >


Thread Tools
Display Modes

  Posting Rules  
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump:
 
  Contains New Posts Forum Contains New Posts   Contains No New Posts Forum Contains No New Posts   A Closed Forum Forum is Closed