pages tagged programmingFeeding the Cloudhttps://feeding.cloud.geek.nz/tags/programming/Feeding the Cloudikiwiki2021-06-11T20:43:57ZProper indentation of Javascript files using js2-mode in emacshttps://feeding.cloud.geek.nz/posts/proper-indentation-of-javascript-files/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2012-05-31T05:30:00Z
<p>If you use <a href="https://www.gnu.org/software/emacs/">emacs</a> for Javascript or <a href="http://nodejs.org/">Node.js</a> development, you should have a look at <a href="https://github.com/mooz/js2-mode">js2-mode</a> (<code>apt-get install js2-mode</code> in <a href="http://packages.debian.org/sid/js2-mode">Debian</a> / <a href="https://launchpad.net/ubuntu/+source/js2-mode">Ubuntu</a>).</p>
<p>In addition to providing syntax highlight, it will parse your Javascript and issue errors and warnings. (It's not as pedantic as <a href="http://www.jslint.com/">jslint</a> or <a href="http://www.jshint.com/">jshint</a> so you'll probably want to fix all of them before committing your files.)</p>
<p>Unfortunately the default indentation style looks a bit crazy and adds excessive indentation to your code. Here's an example:</p>
<pre><code> navigator.id.secret.generate(identity, function (plainKey, wrappedKey) {
var data = {wrappedKey: wrappedKey};
$.post('/loggedin', data, function (res) {
$("#message").text('All done');
}, 'json');
}, function (error) {
$("#message").text('ERROR: ' + error);
});
</code></pre>
<p>It turns out that <a href="http://steve-yegge.blogspot.co.nz/2008_03_01_archive.html">indenting Javascript properly is really hard</a>, but you can turn on a special flag which will cause the indentation to "bounce" between different strategies, starting with the most likely one.</p>
<p>This is what I now have in my <code>~/.emacs</code>:</p>
<pre><code>(custom-set-variables
'(js2-basic-offset 2)
'(js2-bounce-indent-p t)
)
</code></pre>
<p>You can find and configure all other options by issuing the following command:</p>
<blockquote><p>M-x customize-group [RET] js2-mode [RET]</p></blockquote>
<p>So yes, you can have both the reasonable indentation of the standard <a href="http://emacswiki.org/emacs/JavaScriptMode">js-mode</a> and the helpful warnings and errors of <a href="http://emacswiki.org/emacs/Js2Mode">js2-mode</a>!</p>
Code reviews with Gerrit and Gitorioushttps://feeding.cloud.geek.nz/posts/code-reviews-with-gerrit-and-gitorious/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2011-04-17T05:00:00Z
<p>The <a href="http://mahara.org/">Mahara</a> <a href="https://launchpad.net/mahara">project</a> has just moved to mandatory code reviews for every commit that gets applied to core code.</p>
<p>Here is a description of how <a href="http://code.google.com/p/gerrit">Gerrit Code Review</a>, the peer-review system used by <a href="http://www.android.com">Android</a>, was retrofitted into our existing <a href="http://www.git-scm.com/">git</a> repository on <a href="http://www.gitorious.org/">Gitorious</a>.</p>
<p>(If you want to know more about Gerrit, listen to <a href="http://www.twit.tv/floss118">this FLOSS Weekly interview</a>.)</p>
<h3 id="Replacing_existing_Gitorious_committers_with_a_robot">Replacing existing Gitorious committers with a robot</h3>
<p>The first thing to do was to log into Gitorious and remove commit rights from everyone in the <a href="http://www.gitorious.org/mahara/mahara">main repository</a>. Then I created a new <a href="http://www.gitorious.org/~maharabot">maharabot</a> account with a password-less SSH key (stored under <code>/home/gerrit/.ssh/</code>) and made that new account the sole committer.</p>
<p>This is to ensure that nobody pushes to the repository by mistake since all of these changes would be <strong>overwritten by Gerrit</strong>.</p>
<h3 id="Basic_Gerrit_installation">Basic Gerrit installation</h3>
<p>After going through the <a href="https://gerrit.googlecode.com/svn/documentation/2.1.6/install.html">installation instructions</a>, I logged into the Gerrit admin interface and created a new "mahara" <a href="https://gerrit.googlecode.com/svn/documentation/2.1.6/project-setup.html">project</a>.</p>
<p>I picked the "merge if necessary" submit action because "cherry-pick" would <a href="http://groups.google.com/group/repo-discuss/browse_thread/thread/aa53fe63197183af">disable dependency tracking</a> which is quite a handy feature.</p>
<h3 id="Reverse_proxy_using_Nginx">Reverse proxy using Nginx</h3>
<p>Since we wanted to offer Gerrit over HTTPS, I decided to run it behind an <a href="http://www.nginx.org/">Nginx</a> proxy. This is the Nginx configuration I ended up with:</p>
<pre><code>server {
listen 443;
server_name reviews.mahara.org;
add_header Strict-Transport-Security max-age=15768000;
ssl on;
ssl_certificate /etc/ssl/certs/reviews.mahara.org.crt;
ssl_certificate_key /etc/ssl/certs/reviews.mahara.org.pem;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:1m;
ssl_protocols TLSv1;
ssl_ciphers HIGH:!ADH;
ssl_prefer_server_ciphers on;
location / {
proxy_pass http://127.0.0.1:8081;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
}
</code></pre>
<p>Things to note:</p>
<ul>
<li><p>An HTTP to HTTPS redirection is <a href="https://www.owasp.org/index.php/Transport_Layer_Protection_Cheat_Sheet#Rule_-_Do_Not_Perform_Redirects_from_Non-TLS_Page_to_TLS_Login_Page">not provided</a>.</p></li>
<li><p>The <a href="https://secure.wikimedia.org/wikipedia/en/wiki/HTTP_Strict_Transport_Security">HSTS</a> headers indicates to modern browsers that this URL should only be accessed via HTTPS.</p></li>
<li><p>Only strong SSL ciphers are enabled.</p></li>
<li><p>Before proxying requests to Gerrit, Nginx adds a few headers to identify the origin of the request. The <code>Host</code> header in particular must be present otherwise <a href="http://code.google.com/p/gerrit/issues/detail?id=905">built-in Gerrit redirections will fail</a>.</p></li>
</ul>
<h3 id="Mail_setup">Mail setup</h3>
<p>To enable Gerrit to email reviewers and committers, I installed <a href="http://www.postfix.org/">Postfix</a> and used "reviews.mahara.org" as the "System mail name".</p>
<p>Then I added the following to <code>/home/gerrit/mahara_reviews/etc/gerrit.config</code>:</p>
<pre><code>[user]
email = "gerrit@reviews.mahara.org"
</code></pre>
<p>to fix the From address in outgoing emails.</p>
<h3 id="Init_script_and_cron">Init script and cron</h3>
<p>Following the <a href="https://gerrit.googlecode.com/svn/documentation/2.1.6/install.html">installation instructions</a>, I created these symlinks:</p>
<pre><code>ln -s /home/gerrit/mahara_reviews/bin/gerrit.sh /etc/init.d/gerrit
cd /etc/rc2.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc3.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc4.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc5.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc0.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc1.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc6.d && ln -s ../init.d/gerrit K21gerrit
</code></pre>
<p>and put the following settings into <code>/etc/default/gerritcodereview</code>:</p>
<pre><code>GERRIT_SITE=/home/gerrit/mahara_reviews
GERRIT_USER=gerrit
GERRIT_WAR=/home/gerrit/gerrit.war
</code></pre>
<p>to automatically start and stop Gerrit.</p>
<p>I also added a cron job in <code>/etc/cron.d/gitcleanup</code> to ensure that the built-in git repository doesn't get bloated:</p>
<pre>
MAILTO=<i>admin@example.com</i>
20 4 * * * gerrit GIT_DIR=/home/gerrit/mahara_reviews/git/mahara.git git gc --quiet
</pre>
<h3 id="Configuration_enhancements">Configuration enhancements</h3>
<p>To allow images in change requests to be displayed inside the browser, I <a href="http://gerrit.googlecode.com/svn/documentation/2.1.6/config-gerrit.html#mimetype">marked them as safe</a> in <code>/home/gerrit/mahara_reviews/etc/gerrit.config</code>:</p>
<pre><code>[mimetype "image/*"]
safe = true
</code></pre>
<p>Another thing I did to enhance the review experience was to enable the gitweb repository browser:</p>
<pre><code>apt-get install gitweb
</code></pre>
<p>and to make checkouts faster by enabling anonymous Git access:</p>
<pre><code>[gerrit]
canonicalGitUrl = git://reviews.mahara.org/git/
[download]
scheme = ssh
scheme = anon_http
scheme = anon_git
</code></pre>
<p>which requires that you have a git daemon running and listening on port 9418:</p>
<pre><code>apt-get install git-daemon-run
ln -s /home/gerrit/mahara_reviews/git/mahara.git /var/cache/git/
touch /home/gerrit/mahara_reviews/git/mahara.git/git-daemon-export-ok
</code></pre>
<p>Finally, I included the Mahara branding in the header and footer of each page by <a href="http://gerrit.googlecode.com/svn/documentation/2.1.6/config-headerfooter.html#toc0">providing valid XHTML fragments</a> in <code>/home/gerrit/mahara_reviews/etc/GerritSiteHeader.html</code> and <code>GerritSiteFooter.html</code>.</p>
<h3 id="Initial_import_and_replication">Initial import and replication</h3>
<p>Once Gerrit was fully working, I performed the initial code import by using my administrator account to push the exiting Gitorious branches to the internal git repository:</p>
<pre>
git remote add gerrit ssh://<i>username</i>@reviews.mahara.org:29418/mahara
git push gerrit 1.2_STABLE
git push gerrit 1.3_STABLE
git push gerrit master
</pre>
<p>Note that I had to <strong>temporarily disable "Require Change IDs"</strong> in the project settings in order to import the old commits which didn't have these.</p>
<p>To replicate the internal Gerrit repository back to Gitorious, I created a new <code>/home/gerrit/mahara_reviews/etc/replication.config</code> file:</p>
<pre><code>[remote "gitorious"]
url = gitorious.org:mahara/${name}.git
push = +refs/heads/*:refs/heads/*
push = +refs/tags/*:refs/tags/*
</code></pre>
<p>(The ${name} variable is <a href="https://code.google.com/p/gerrit/issues/detail?id=896">required</a> even when you have a single project.)</p>
<h3 id="Contributor_instructions">Contributor instructions</h3>
<p>This is how developers can get a working checkout of our code now:</p>
<pre>
git clone git://gitorious.org/mahara/mahara.git
cd mahara
git remote add gerrit ssh://<i>username</i>@reviews.mahara.org:29418/mahara
git fetch gerrit
scp -p -P 29418 reviews.mahara.org:hooks/commit-msg .git/hooks/
</pre>
<p>and this is how they can submit local changes to Gerrit:</p>
<pre><code>git push gerrit HEAD:refs/for/master
</code></pre>
<p>Anybody can submit change requests or <a href="https://reviews.mahara.org">comment on them</a> but make sure you <strong>do not have the Cookie Pie</strong> Firefox extension installed or you will be <a href="https://code.google.com/p/gerrit/issues/detail?id=887">unable to log into Gerrit</a>.</p>
Using plupload inside a Django applicationhttps://feeding.cloud.geek.nz/posts/using-plupload-inside-django/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2011-04-07T20:00:00Z
<p><a href="http://plupload.org/">Plupload</a> is a reusable component which fully takes advantage of the file upload capabilities of your browser and its plugins. It will use HTML5, Flash and Google Gears if they are available, but otherwise, it can gracefully degrade down to HTML4 it it needs to. Here's how it can be integrated within a Django web application.</p>
<p>(I have posted the <a href="https://github.com/fmarier/plupload_django">sample application</a> I will refer to and you may use it <a href="https://github.com/fmarier/plupload_django/blob/master/COPYING.txt">anyway you like</a>.)</p>
<h3 id="Creating_a_basic_upload_form">Creating a basic upload form</h3>
<p>The first step is to create a simple one-file upload form that will be used in the case where Javascript is disabled:</p>
<pre><code>class UploadForm(forms.Form):
file = forms.FileField()
def save(self, uploaded_file):
print 'File "%s" would presumably be saved to disk now.' % uploaded_file
pass
</code></pre>
<p>Then you can add this form to one of your templates:</p>
<pre><code><form enctype="multipart/form-data" action="{% url plupload_sample.upload.views.upload_file %}" method="post">
{% csrf_token %}
<div id="uploader">
{{form.file.errors}}{{form.file}}
<input type="submit" value="Upload" />
</div>
</form>
</code></pre>
<p>And create a new method to receive the form data:</p>
<pre><code>@csrf_protect
def upload_file(request):
if request.method == 'POST':
form = UploadForm(request.POST, request.FILES)
if form.is_valid():
uploaded_file = request.FILES['file']
form.save(uploaded_file)
return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))
else:
form = UploadForm()
return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))
</code></pre>
<h3 id="Adding_Plupload_to_the_template">Adding Plupload to the template</h3>
<p>In order to display the right Javascript-based upload form, add the following code, based on the <a href="http://plupload.org/example_queuewidget.php">official example</a>, to the head of your template:</p>
<pre><code><link rel="stylesheet" href="/css/plupload.queue.css" type="text/css">
<script type="text/javascript" src="/js/jquery.min.js"></script>
<script type="text/javascript" src="/js/plupload.full.min.js"></script>
<script type="text/javascript" src="/js/jquery.plupload.queue.min.js"></script>
<script type="text/javascript">
$(function() {
$("#uploader").pluploadQueue({
runtimes : 'html5,html4',
url : '{% url plupload_sample.upload.views.upload_file %}',
max_file_size : '1mb',
chunk_size: '1mb',
unique_names : false,
multipart: true,
headers : {'X-Requested-With' : 'XMLHttpRequest', 'X-CSRFToken' : '{{csrf_token}}'},
});
$('form').submit(function(e) {
var uploader = $('#uploader').pluploadQueue();
// Validate number of uploaded files
if (uploader.total.uploaded == 0) {
// Files in queue upload them first
if (uploader.files.length > 0) {
// When all files are uploaded submit form
uploader.bind('UploadProgress', function() {
if (uploader.total.uploaded == uploader.files.length)
$('form').submit();
});
uploader.start();
} else {
alert('You must at least upload one file.');
}
e.preventDefault();
}
});
});
</script>
</code></pre>
<p>Pay close attention to the extra headers that need to be added to ensure that the AJAX requests will pass the <a href="http://docs.djangoproject.com/en/1.3/ref/contrib/csrf/">Django CSRF checks</a>:</p>
<pre><code>X-Requested-With: XMLHttpRequest
X-CSRFToken: {{csrf_token}}
</code></pre>
<h3 id="Adding_Plupload_to_the_view_method">Adding Plupload to the view method</h3>
<p>Now in order to properly receive the files uploaded by Plupload via AJAX calls, we need to revise our upload_file() method:</p>
<pre>
@csrf_protect
def upload_file(request):
if request.method == 'POST':
form = UploadForm(request.POST, request.FILES)
if form.is_valid():
uploaded_file = request.FILES['file']
form.save(uploaded_file)
<b>if request.is_ajax():
response = HttpResponse('{"jsonrpc" : "2.0", "result" : null, "id" : "id"}', mimetype='text/plain; charset=UTF-8')
response['Expires'] = 'Mon, 1 Jan 2000 01:00:00 GMT'
response['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
response['Pragma'] = 'no-cache'
return response
else:
return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))</b>
else:
form = UploadForm()
return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))
</pre>
<p>The above includes the response (which is not really documented as far as I can tell) that needs to be sent back to Plupload to make sure it knows that the file has been received successfully:</p>
<pre><code>{"jsonrpc" : "2.0", "result" : null, "id" : "id"}
</code></pre>
<h3 id="Adding_support_for_multipart_files">Adding support for multipart files</h3>
<p>Our solution so far works fine except when uploading large files that need to be sent in multiple chunks.</p>
<p>This involves writing to a temporary file until all parts have been received:</p>
<pre>
@csrf_protect
def upload_file(request):
if request.method == 'POST':
form = UploadForm(request.POST, request.FILES)
if form.is_valid():
uploaded_file = request.FILES['file']
<b>chunk = request.REQUEST.get('chunk', '0')
chunks = request.REQUEST.get('chunks', '0')
name = request.REQUEST.get('name', '')
if not name:
name = uploaded_file.name
temp_file = '/tmp/insecure.tmp'
with open(temp_file, ('wb' if chunk == '0' else 'ab')) as f:
for content in uploaded_file.chunks():
f.write(content)
if int(chunk) + 1 >= int(chunks):
form.save(temp_file, name)</b>
if request.is_ajax():
response = HttpResponse('{"jsonrpc" : "2.0", "result" : null, "id" : "id"}', mimetype='text/plain; charset=UTF-8')
response['Expires'] = 'Mon, 1 Jan 2000 01:00:00 GMT'
response['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
response['Pragma'] = 'no-cache'
return response
else:
return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))
else:
form = UploadForm()
return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))
</pre>
<p>Note that I have used <code>/tmp/insecure.tmp</code> for brevity. In a real application, you do need to use a <a href="http://docs.python.org/library/tempfile.html">secure mechanism</a> to create the temporary file or you would expose yourself to a <a href="https://duckduckgo.com/?q=insecure+temp+file+creation">tempfile vulnerability</a>.</p>
Handling email-based events safelyhttps://feeding.cloud.geek.nz/posts/handling-email-based-events-safely/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2010-06-14T05:20:00Z
<p>Mahara.org users have recently witnessed the unfortunate effects of a bug in the Mahara event handling: the Mahara cron job got stuck in an email loop and kept on sending the <a href="http://mahara.org/interaction/forum/topic.php?id=1882">same forum post</a> over and over again.</p>
<p>Here is what the affected code used to look like (edited for clarity):</p>
<pre><code>db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
handle_activity($activity);
}
delete_records('activity_queue');
db_commit();
</code></pre>
<p>One of the problems with this code was that if the <code>handle_activity()</code> function threw an exception, it would interfere with the processing of the entire queue. So <a href="https://github.com/MaharaProject/mahara/commit/823ab3168aed5ae0e4de27139d56b50c0ab0787c">it got fixed</a> in the following way:</p>
<pre>
db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
<b>try {</b>
handle_activity($activity);
<b>}
catch (MaharaException $e) {
log_debug($e->getMessage());
}</b>
}
delete_records('activity_queue');
db_commit();
</pre>
<p>Much better. However, there was still a problem with the code: the whole queue processing is contained within a transaction. This means that should any of the SQL statements fail at any point, the exception would be caught but the SQL statements would all be rolled back at the end by the database.</p>
<p>Now the idea of a transaction is good: we want activity handling to either succeed entirely or be rolled back. But the fact that some activities cannot be handled should not interfere with other activities. So this has <a href="https://github.com/MaharaProject/mahara/commit/d08395ac1308727dd45396083f454c47605c0604">been fixed</a> by moving the transaction to the inside of the loop:</p>
<pre>
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
<b>db_begin();</b>
try {
handle_activity($activity);
}
catch (MaharaException $e) {
log_debug($e->getMessage());
}
<b>db_commit();</b>
}
delete_records('activity_queue');
</pre>
<p>So individual activites are allowed to fail and get rolled back, but they will not affect other activites. But there was still one remaining problem: what if we encounter an error we cannot catch? For example, what would happen if PHP were to segfault or run out of memory while the activity queue is being processed?</p>
<p>Well, in that case, it turns out that Mahara will never reach the <code>delete_records()</code> call and the activity queue will never be cleared. Which means that on the next cron run, all of the activities will be handled again, even the ones that were successfully handled already (i.e. emails will be sent over and over again).</p>
<p>The way we <a href="https://github.com/MaharaProject/mahara/commit/d736fd6c0c99cb998af56979b12ceab60ed101ee">fixed this problem</a> was by moving the <code>delete_records()</code> from the end of the function to the beginning of the loop:</p>
<pre>
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
<b>delete_records('activity_queue', 'id', $activity->id);</b>
db_begin();
try {
handle_activity($activity);
}
catch (MaharaException $e) {
log_debug($e->getMessage());
}
db_commit();
}
</pre>
<p>Each activity is removed from the queue <strong>before</strong> it is processed.</p>
<p>Unfortunately, there is a downside to this modification: should an activity handler fail for whatever reason, no further attempts will be made. This means that some notifications could be lost if an unexpected error occurs.</p>
<p>However, given that some of the activity handlers send emails out into the world and that it is not possible to "un-send" them, this is the only way we can guarantee that no duplicate emails will be sent. Of course, if you notice that certain notifications are lost because of a bug in Mahara, <a href="https://bugs.launchpad.net/mahara/+filebug">let us know</a> and we'll fix it!</p>