2

I am new to the whole distributed computing / cloud thing. But I had an idea at work for our multimedia stuff like movie encoding / cpu intensive things tasks (which sometimes take a few hours).

Is there a 'free' (linux?) way to go about using a Windows machine, and offsetting those cpu cycles for that task to say 10 servers that are generally idle (cpu wise)?

I'm just curious if there is a way to do this or am I just grasping at straws here. My thought is that a 'cloud' setup would achieve this, however like I stated initially, I am a total newbie when it comes to it.

This is just an idea, looking for some thoughts? Anyone achieve this?

Improve this question

4 Answers 4

Reset to default
3

Corporately we use grid computing for things similar to this. A great one to investigate is Condor.

Improve this answer
1
  • Also check out Sun/Oracle Grid Engine (similar product), available as open source too at gridengine.sunsource.net Commented May 4, 2010 at 15:03
2

are those 'cpu intensive things'...

  • developed in-house? then you'll have to split and distribute the job yourself. there are some nice libraries, but everything is very low level.
  • prebuilt software? then talk to the developer/provider. either it's supported or it's not
  • same process applied to lots of individual items (for example, processing thousands of images)? then you might be able to set a simple system out of a DB table holding the subtask status + some batch scripts that: pick a task, mark as doing, do it, return the result, mark the task as done, repeat until all tasks are done.
Improve this answer
1

I think that if you are talking about render farms, there are some examples of such projects.

Improve this answer
2
  • Not just render farms, but more like for large cpu intensive cycles, somehow 'funneling' that work through a server farm, but one that is not dedicated just for rendering. Not sure if I make sense... :P Commented Apr 30, 2010 at 17:12
  • sure it makes sense. you're thinking in terms of mosix and beowulf clusters! but i think that application supported distribution is better than a generic solution. Commented May 1, 2010 at 15:06
1

Ubuntu with Sun Grid Engine would work and is available in the repos. I think everyone's ideas are going to require you to wipe the host servers though, not sure if that's what you're looking for.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.