Crowd Programming Enginehttp://crowdprogramming.postach.io/feed.xml2020-11-13T13:46:58.233000ZWerkzeugThespian Python Actorshttps://crowdprogramming.postach.io/post/thespian-python-actors2020-11-13T13:46:58.233000Z2020-11-13T13:43:44ZCharles Kosta, Sc.D.<div><br /></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"><a href="https://thespianpy.com/doc/">Thespian </a>is a Python library providing a framework for developing concurrent, distributed, fault tolerant applications. </span></span></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"> Thespian is built on the <a href="https://wikipedia.org/wiki/Actor_model" rev="en_rl_none"><span style="color:rgb(32, 97, 162);">Actor Model</span></a> which allows applications to be written as a group of independently executing but cooperating "Actors" which communicate via messages. These Actors run within the Actor System provided by the Thespian library. </span></span></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"><span style="--en-markholder:true;"><br /></span></span></span></div>
<div style="color:rgb(21, 114, 174);"><b><span style="font-family: Actor;"><span style="font-size: 100%;">Concurrent</span></span></b></div>
<div style="padding-left:40px;"><span style="font-family: Actor;"><span style="font-size: 100%;">All Actors run independently within the Actor System. The Actor System may run the Actors as threads, processes, or even sequential operations within the current process—all with no change to the Actors themselves.</span></span></div>
<div><br /></div>
<div style="color:rgb(21, 114, 174);"><b><span style="font-family: Actor;"><span style="font-size: 100%;">Distributed</span></span></b></div>
<div style="padding-left:40px;"><span style="font-family: Actor;"><span style="font-size: 100%;">Actors run independently…anywhere. Multiple servers can each be running Thespian and an Actor can be run on any of these systems—all with no change to the Actors themselves. Thespian handles the communication between the Actors and the management process of distributing the Actors across the systems.</span></span></div>
<div><br /></div>
<div style="color:rgb(21, 114, 174);"><b><span style="font-family: Actor;"><span style="font-size: 100%;">Location Independent</span></span></b></div>
<div style="padding-left:40px;"><span style="font-family: Actor;"><span style="font-size: 100%;">Because Actors run independently anywhere, they run independently of their actual location. A distributed Actor application may have part of it running on a local server, part running on a server in Amsterdam, and part running on a server in Singapore… or not, with no change or awareness of this by the Actors themselves.</span></span></div>
<div><br /></div>
<div style="color:rgb(21, 114, 174);"><b><span style="font-family: Actor;"><span style="font-size: 100%;">Fault Tolerant</span></span></b></div>
<div style="padding-left:40px;"><span style="font-family: Actor;"><span style="font-size: 100%;">Individual Actors can fail and be restarted—automatically—without impact to the rest of the system.</span></span></div>
<div><br /></div>
<div style="color:rgb(21, 114, 174);"><b><span style="font-family: Actor;"><span style="font-size: 100%;">Scalable</span></span></b></div>
<div style="padding-left:40px;"><span style="font-family: Actor;"><span style="font-size: 100%;">The number of Actors in the system can be dynamically extended based on factors such as work volume, and systems added to the Distributed Actor System environment are automatically utilized.</span></span></div>
<div><br /></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"> One of the key aspects of the Actor Model is that it represents a higher level of abstraction than is provided by most frameworks. When writing an Actor-based application, the concurrency and transport layers are completely abstracted, which both simplifies the design and allows the concurrency or transport to be changed in the future <i><span style="color:rgb(26, 26, 26);">without</span></i> requiring changes in the Actor-based application. </span></span></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"> The above qualities of Actor programming make it ideally suited for Cloud-based applications as well, where compute nodes are added and removed from the environment dynamically.</span></span></div>
<div><span style="font-family: Actor;"><span style="font-size: 100%;"><span style="--en-markholder:true;"><br /></span></span></span></div><pre><div>History</div>
<div>-------</div>
<div><br /></div>
<div>Thespian was developed at GoDaddy as part of the support for GoDaddy's</div>
<div>dedicate and virtual server product line. At one point, Thespian was</div>
<div>helping to manage 10,000 physical servers via a combination of</div>
<div>on-server and remote actors.</div>
<div><br /></div>
<div>The primary Thespian author (Kevin Quick) is no longer working at</div>
<div>GoDaddy, but through GoDaddy's willingness to open-source the Thespian</div>
<div>code and transfer the rights to this author, this repository is still</div>
<div>being maintained and developed. With heartfelt thanks to the Vertigo</div>
<div>team at GoDaddy, this Python Actor model has been developed and</div>
<div>refined to provide a highly functional library for general use.</div></pre><div><br /></div>
Getting Parallel with PyClipshttps://crowdprogramming.postach.io/post/getting-parallel-with-pyclips2020-11-12T02:29:44.160000Z2020-11-12T02:26:11ZCharles Kosta, Sc.D.<div>This module embeds a fully functional CLIPS engine into Python, and gives the developer a more Python-compliant interface to CLIPS without cutting down on functionalities. In fact CLIPS is compiled into the module in its entirety, and most API functions are bound to Python methods. However the direct bindings to the CLIPS library (implemented as the <tt>_clips</tt> submodule) are not described here: each function is described by an appropriate documentation string, and accessible by means of the <tt>help()</tt> function or through the <b>pydoc</b> tool. Each direct binding maps to an API provided function. For a detailed reference<span style="font-size: 11.6667px;"> </span>for these functions see <i>Clips Reference Guide Vol. II: Advanced Programming Guide</i>, available for download at the CLIPS website.</div>
<div><tt>PyCLIPS</tt> is also capable of generating CLIPS text and binary files: this allows the user to interact with sessions of the CLIPS system itself. An important thing to know, is that <tt>PyCLIPS</tt> implements CLIPS as a separated<span style="font-size: 11.6667px;"> </span>engine: in the CLIPS module implementation, CLIPS ``lives'' in its own memory space, allocates its own objects. The module only provides a way to send information and commands to this engine and to retrieve results from it.</div>
<div><br /></div>
<div><a href="http://pyclips.sourceforge.net/manual/" rev="en_rl_none" shape="rect">http://pyclips.sourceforge.net/manual/</a></div>
Crowd Programming Engine (CPE)https://crowdprogramming.postach.io/post/crowd-programming-engine-cpe-22020-10-14T11:57:06.797000Z2020-09-21T18:10:39ZCharles Kosta, Sc.D.<div>"A Distributed Agent-Based system with PyClips, RandomForests and, ThespianPy Actors over a shared Tuple-Space." - Charles Kosta, Sc.D.</div>
Getting Started with NetWorkSpaceshttps://crowdprogramming.postach.io/post/getting-started-with-networkspaces2018-04-24T14:36:38.538000Z2018-04-21T13:02:14ZCharles Kosta, Sc.D.<div>We are ultimately trying to use NetWorkSpaces (NWS) as a way for Python, R, and Java agents running on different machines to<br clear="none"/>communicate and coordinate with one another. <br clear="none"/><br clear="none"/>To get started with Python NWS, we'll first have to install the NWS server, and the Python NWS client.</div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>
<div><span>pip install nwsclient (<a shape="rect" href="https://pypi.org/project/nwsclient/1.6.0/" target="_blank">https://pypi.org/project/nwsclient/1.6.0/</a>)</span></div>
<div><br clear="none"/></div>
<div><span>pip install nwsserver</span> (<a shape="rect" href="https://pypi.org/project/nwsserver/" target="_blank">https://pypi.org/project/nwsserver/</a>)</div>
<div><br clear="none"/></div>
<div><span>pip install Twisted</span> (<a shape="rect" href="https://pypi.org/project/Twisted/" target="_blank">https://pypi.org/project/Twisted/</a>)</div>
<div><br clear="none"/><br clear="none"/>Next, an NWS server must be started. This can be done from a shell<br clear="none"/>using the nws command, as follows:<br clear="none"/><br clear="none"/>% twistd -ny .local/nws.tac <br clear="none"/><br clear="none"/>From another window, start an interactive Python session, and import<br clear="none"/>the NetWorkSpace class:<br clear="none"/><br clear="none"/>% python<br clear="none"/>>>> from nws.client import NetWorkSpace<br clear="none"/><br clear="none"/>Next, create a work space called 'cpe':<br clear="none"/><br clear="none"/>>>> ws = NetWorkSpace('cpe')<br clear="none"/><br clear="none"/>This is using the NWS server on the local machine. Additional arguments<br clear="none"/>can be used to specify the hostname and port used by the NWS server if<br clear="none"/>necessary.<br clear="none"/><br clear="none"/>Once we have a work space, we can write data into a variable using the<br clear="none"/>store method:<br clear="none"/><br clear="none"/>>>> ws.store('joe', 17)<br clear="none"/><br clear="none"/>The variable 'joe' now has a value of 17 in our work space.<br clear="none"/><br clear="none"/>To read that variable we use the find method:<br clear="none"/><br clear="none"/>>>> age = ws.find('joe')<br clear="none"/><br clear="none"/>which sets 'age' to 17.<br clear="none"/><br clear="none"/>Note that the find method will block until the variable 'joe' has a<br clear="none"/>value. That is important when we're trying to read that variable from a<br clear="none"/>different machine. If it didn't block, you might have to repeatedly try<br clear="none"/>to read the variable until it succeeded. </div>
<div><br clear="none"/></div>
<div>There are times when you don't want to block, but just want to see if </div>
<div>some variable has a value. That is done with the <strong>findTry</strong> method.</div>
<div><br clear="none"/>--------------------------------------------------------------------------------------<br clear="none"/></div>
<div><br clear="none"/></div>
<div>From another window, start an interactive Python session, and import<br clear="none"/>the NetWorkSpace class:<br clear="none"/><br clear="none"/>% python<br clear="none"/>>>> from nws.client import NetWorkSpace<br clear="none"/><br clear="none"/>Next, connect to a work space called 'cpe':<br clear="none"/><br clear="none"/>>>> ws = NetWorkSpace('cpe')<br clear="none"/></div>
<div><br clear="none"/></div>
<div>This won't create the 'cpe' work space, since it already exists. Now you<br clear="none"/>can execute a operation such as:<br clear="none"/><br clear="none"/>>>> x = ws.fetch('joe')</div>
<div>>>> x</div>
<div>17</div>
<div><br clear="none"/>you should see that you got the value 17 from the previous w.store() function;</div>
<div>only this time you fetched it permanently. Now try to execute the the fetch again.</div>
<div><br clear="none"/></div>
<div>>>> x = ws.fetch('joe')<br clear="none"/></div>
<div><br clear="none"/></div>
<div>Your program will now be blocked! in this session, watch it block for a minute, and then execute a 'store' in</div>
<div>the other session:<br clear="none"/><br clear="none"/></div>
<div>>>> ws.store('joe', 18)<br clear="none"/></div>
<div><br clear="none"/></div>
<div>You should see the previous session return from the fetch command and x should now be 18.</div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>
<div>That's a basic sample of this technology, which of course can become much more complex. Keep in mind</div>
<div>that these programs could have been on different machines; or hosted on Amazon. </div>
<div><br clear="none"/></div>
<div>More importantly, an R program and a python program</div>
<div>and even a Java program could be working cooperatively.</div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>
NetWorkSpaces for Pythonhttps://crowdprogramming.postach.io/post/networkspaces-for-python2018-04-21T22:53:55.066000Z2018-04-18T17:20:04ZCharles Kosta, Sc.D.<div><h1><span>NetWorkSpaces: a framework to coordinate distributed programs</span></h1></div><div><div><p><br clear="none"/></p><p>NetWorkSpaces (NWS) is one way to write parallel programs. It allows you to take advantage of a multi-core machine; multiple virtual machines, as well as cloud-based clusters, using languages such as <span style="text-decoration: underline;"><strong>Python, R, Java, and Matlab</strong></span>. With NetWorkSpaces for Python, you can execute Python functions and programs in parallel using methods very much like the standard Python <strong>map</strong> function. In some cases, you may be able to parallelize your program in minutes, rather than months.</p><p>For example, here's a simple Python NWS script:</p><pre xml:space="preserve"> from math import sqrt
from nws.sleigh import Sleigh
s = Sleigh()
for x in s.imap(sqrt, xrange(10)):
print x
</pre><p>It looks pretty simple, but you'll need to be familiar with the <strong>imap</strong> function in the standard <strong>itertools</strong> module.</p></div></div><div><br clear="none"/></div>
<div><a shape="rect" href="http://nws-py.sourceforge.net/" target="_blank">http://nws-py.sourceforge.net/</a><br clear="none"/></div>
<div><a shape="rect" href="https://github.com/bigcomputing/big-server" target="_blank">https://github.com/bigcomputing/big-server</a><br clear="none"/></div>
<div><a shape="rect" href="http://nws-r.sourceforge.net/" target="_blank">http://nws-r.sourceforge.net/</a><br clear="none"/></div>
<div><br clear="none"/></div>
<div><br clear="none"/></div>