[gpars-dev] OFFTOPIC: GPars performance vs Go [was Re: A few notes on choosing between Go and D for a quick project]

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gpars-dev] OFFTOPIC: GPars performance vs Go [was Re: A few notes on choosing between Go and D for a quick project]

Russel Winder-2
On Tue, 2015-03-17 at 20:50 +0000, Bienlein via Digitalmars-d wrote:

> > Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP,
> > not to
> > mention JCSP and GPars.
>
> I'm not really sure whether this can be put exactly that way. On
> a machine with 4 GB RAM you can spawn about 80.000 goroutines
> (aka green threads). Let's say each threads calculates a large
> fibonacci number. If the fibonacci calculation yields the
> processor frequently all 80.000 run seamingly in parallel and
> return their result almost "at the same time".
>
> With GPars this would not work. You can only start some 2.000
> Java threads on a machine with the same amount of memory. If also
> the fibonacci calculation in Groovy/GPars yields the processor,
> using GPars the first 2.000 fibonacci calculations will
> nevertheless crowd out the following 2.000 threads of all those
> 80.000 fibonacci calculations and so on. I once tried this out
> with both Go and Groovy/GPars.
<This needs to go to the GPars mailing lists list so I have cross
posted.>

A priori I do not believe the claim made here: a GPars task is submitted
to a thread pool, which is exactly what the goroutines are. Thus the
number of Java threads is not a bound on the number of GPars tasks. Any
bounds will be provided by the Fork/Join pool.

If you have code you can send me, I will run it and do the benchmarking.
If not I guess I will have to write some. Until we have the results of
executing code, the above must be treated as pure conjecture, which I
believe will be refuted. If the claim however turns out to be true, then
it highlights a bug in GPars.

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gpars-dev] OFFTOPIC: GPars performance vs Go [was Re: A few notes on choosing between Go and D for a quick project]

Vaclav
Administrator
You are correct, Russel.
Only if using the default pool, which resizes with the number of tasks in the queue, you could observe an increasing number of threads.

Vaclav


On Wed, Mar 18, 2015 at 9:40 AM, Russel Winder <[hidden email]> wrote:
On Tue, 2015-03-17 at 20:50 +0000, Bienlein via Digitalmars-d wrote:
> > Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP,
> > not to
> > mention JCSP and GPars.
>
> I'm not really sure whether this can be put exactly that way. On
> a machine with 4 GB RAM you can spawn about 80.000 goroutines
> (aka green threads). Let's say each threads calculates a large
> fibonacci number. If the fibonacci calculation yields the
> processor frequently all 80.000 run seamingly in parallel and
> return their result almost "at the same time".
>
> With GPars this would not work. You can only start some 2.000
> Java threads on a machine with the same amount of memory. If also
> the fibonacci calculation in Groovy/GPars yields the processor,
> using GPars the first 2.000 fibonacci calculations will
> nevertheless crowd out the following 2.000 threads of all those
> 80.000 fibonacci calculations and so on. I once tried this out
> with both Go and Groovy/GPars.

<This needs to go to the GPars mailing lists list so I have cross
posted.>

A priori I do not believe the claim made here: a GPars task is submitted
to a thread pool, which is exactly what the goroutines are. Thus the
number of Java threads is not a bound on the number of GPars tasks. Any
bounds will be provided by the Fork/Join pool.

If you have code you can send me, I will run it and do the benchmarking.
If not I guess I will have to write some. Until we have the results of
executing code, the above must be treated as pure conjecture, which I
believe will be refuted. If the claim however turns out to be true, then
it highlights a bug in GPars.

--
Russel.
=============================================================================
Dr Russel Winder      t: <a href="tel:%2B44%2020%207585%202200" value="+442075852200">+44 20 7585 2200   voip: [hidden email]
41 Buckmaster Road    m: <a href="tel:%2B44%207770%20465%20077" value="+447770465077">+44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [gpars-dev] OFFTOPIC: GPars performance vs Go [was Re: A few notes on choosing between Go and D for a quick project]

Kerridge, Jon
Hi,
I too have been 'playing with GO' and yes you can set of a lot of goroutines BUT and it is a big BUT,
when I last tried there was a problem.

Essentially, regardless of the underlying thread model a go routine is like a fork; unfortunately there is no equivalent of join.  Thus you cannot set a bunch of go routines running and then wait until all of them have returned or completed before going on to the next step of an algorithm.

This is a fundamental requirement of any CSP based model and is implemented in all the other models, which all implement the join phase.  The problem manifests itself if you construct an algorithm that assumes the system waits until all go routines initiated together wait until they all terminate before proceeding.  The scheduler generates a whole load of go routines waiting to input on a channel and then runs out of memory.  I raised a bug report etc.

I then had long email discussions with the  designer of the go scheduler who could not see my point, even though there are papers outlining the problem from a number of different users and institutions including CMU.  I found the go guy's attitude somewhat frustrating.

Jon

Professor Jon Kerridge
School of Computing
Edinburgh Napier University
Merchiston Campus
Edinburgh EH10 5DT
 
0131 455 2777
 

From: Václav Pech [[hidden email]]
Sent: 18 March 2015 09:49
To: GParS Developers
Cc: digitalmars.D; GPars_Users
Subject: Re: [gpars-dev] OFFTOPIC: GPars performance vs Go [was Re: A few notes on choosing between Go and D for a quick project]

You are correct, Russel.
Only if using the default pool, which resizes with the number of tasks in the queue, you could observe an increasing number of threads.

Vaclav


On Wed, Mar 18, 2015 at 9:40 AM, Russel Winder <[hidden email]> wrote:
On Tue, 2015-03-17 at 20:50 +0000, Bienlein via Digitalmars-d wrote:
> > Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP,
> > not to
> > mention JCSP and GPars.
>
> I'm not really sure whether this can be put exactly that way. On
> a machine with 4 GB RAM you can spawn about 80.000 goroutines
> (aka green threads). Let's say each threads calculates a large
> fibonacci number. If the fibonacci calculation yields the
> processor frequently all 80.000 run seamingly in parallel and
> return their result almost "at the same time".
>
> With GPars this would not work. You can only start some 2.000
> Java threads on a machine with the same amount of memory. If also
> the fibonacci calculation in Groovy/GPars yields the processor,
> using GPars the first 2.000 fibonacci calculations will
> nevertheless crowd out the following 2.000 threads of all those
> 80.000 fibonacci calculations and so on. I once tried this out
> with both Go and Groovy/GPars.

<This needs to go to the GPars mailing lists list so I have cross
posted.>

A priori I do not believe the claim made here: a GPars task is submitted
to a thread pool, which is exactly what the goroutines are. Thus the
number of Java threads is not a bound on the number of GPars tasks. Any
bounds will be provided by the Fork/Join pool.

If you have code you can send me, I will run it and do the benchmarking.
If not I guess I will have to write some. Until we have the results of
executing code, the above must be treated as pure conjecture, which I
believe will be refuted. If the claim however turns out to be true, then
it highlights a bug in GPars.

--
Russel.
=============================================================================
Dr Russel Winder      t: <a href="tel:%2B44%2020%207585%202200" value="&#43;442075852200" target="_blank"> +44 20 7585 2200   voip: [hidden email]
41 Buckmaster Road    m: <a href="tel:%2B44%207770%20465%20077" value="&#43;447770465077" target="_blank"> +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



--
Loading...