technotroph

random thoughts about mozilla hacking and the computing world

Fiddling with the OCaml Type System

Not too long ago, in OCaml 4.00, the guys working on OCaml exposed the compiler internals and packaged it as a findlib library. The library allows people working on editor tools to get access to the type system, the ast structures, the typed tree structures and many other cool stuff. One of the things you can do with this library is that you can get the types of a piece of code. Yep, it gives you some form of reflection capabilities. How cool is that?! One of the use case of this is that it allows programmers to use camlp4 to figure which printer to use when printing values, it gives some form of generic printing.

Some of the modules that are required to get the types of the code are TypedTree, Types, Compmisc, Lexing, Parse and Printtyp. I will go through each of these later.

Firstly, we will need to get the initial environment (Env type) to work on. We need this when we are inferring the code. We can get this environment from Compmisc.

let e =
Compmisc.init_path false;
Compmisc.initial_env ()

Now, let’s get the AST of the code that we want to infer the types from. We will need to do lexical analyzing on it and parse the tokens. The good news is that all these are provided in the compiler-libs as functions. The tokens and AST are just a function call away. Say, we want to get the types of this add function:

let add x y = x + y

We can first pass the string representation of the code to the Lexer, and then pipe it (with the |> operator) to the parser.

let ast = "let add x y = x + y" |> Lexing.from_string |> Parse.implementation

Note that the (|>) operator is introduced in 4.01.0 of OCaml. You will need to upgrade your compiler to use it. You can also define it this way (but it may not be as efficient) :

let (|>) x f = f x

Now that we have the ast, we can now get the types of the ast. We will need to use the type_structure function from the Typemod module to do this. Simply pass the ast and the initial environment to the function.

let tstr, _tsig, _newe = Typemod.type_structure env ast Location.none

The return value is a triple with the type structure (a type representation of the expression’s type), the type signature and the new environment after we have inferred the types. We will work will the type structure and ignore the rest.
After some destructuring and unwrapping, you should get the types:

let {str_items} = tstr in
let {str_desc} :: _ = str_items in
let Typedtree.Tstr_value (_, lst) = str_desc in
let rec helper = function
| [] -> ()
| ({ pat_type }, _) :: rest ->
Printtyp.raw_type_expr Format.std_formatter pat_type;
helper rest in
helper lst;

Finally, to compile the code, you need to compile it with the compiler-libs package and link it with the ocamlcommon library.

ocamlbuild -pkg compiler-libs -lib ocamlcommon gettyp.ml

That’s it. The code in this post is found in this gist. Happy hacking!

Talos Mozharness is Now Integrated with Mach

Testing Talos and testing how it’s run in production machines has gotten much easier! I have recently wrote a patch for a mach command that integrates Talos Mozharness into it. This patch clones mozharness and runs the talos_script.py in mozharness to run Talos and Talos tests on the compiled binary that is in the tree.

Running Talos

First, make sure that firefox is already compiled. Then, execute the mach command in the top source directory. Let’s try running the chromez test suite. The list of valid suites are in the talos.json file.

$ ./mach talos-test chromez

And…that’s it! Mozharness will be cloned into the obj-dir and you can see the output from Talos soon after.

Problems?

If there are any issues with the command please file a bug and CC me to the bug. Enjoy!

Tracing JIT Compiler in PyPy

Typically tracing VMs go through various phases when they execute a program: Interpretation/profiling, tracing, code generation and execution of the generated code. When the program starts, everything is interpreted. The inter- preter does lightweight profiling to establish which loops are run most frequently. This lightweight profiling is usually done by having a counter on each backward jump instruc- tion that counts how often this particular backward jump is executed. Since loops need a backward jump somewhere, this method looks for loops in the user program.

Taken from a paper about PyPy’s tracing JIT written by the developers of PyPy. The paper explains the technique behind the tracing JIT very well. Read it if you want to know the gory details behind PyPy.

Moving Talos from BuildBot to Mozharness

tl;dr Using BuildBot to run Talos is giving us too many headaches. We are moving it to Mozharness, a configuration driven script harness with full logging facilities that allows production infrastructure and individual developers to use the same scripts.

In case you don’t know, I’m currently interning at Mozilla as a release engineer in the RelEng team. Part of my job here is to port Talos from BuildBot to Mozharness. Buildbot is a continuous integration tool, kinda like Travis-CI and Jekins-CI. It is used at mozilla to automate our builds and tests. Talos, is a performance testing framework that loads a number of test pages and collect data from the system. It is usually scheduled by BuildBot to be ran after the builds have been completed.

Talos Py

Being called and ran by BuildBot has a number of downsides. The biggest problem is that other developers have to understand and setup buildbot on their own dev machines to test the production infrastructure environment that talos runs in.

But with mozharness developers just have to clone the mozharness code and execute it to test talos. Assuming that you wish to run the tp5o talos test suite and your dev machine is a 64 bit OS X machine, you can run these commands to start talos.

hg clone http://hg.mozilla.org/build/mozharness
cd mozharness
python scripts/talos_script.py --suite tp5o --add-option --webServer,localhost --branch-name Ash --system-bits 64 --cfg talos/mac_config.py --download-symbols ondemand --use-talos-json

The best part is that developers just have to copy and paste the commands that are logged in TBPL to test particular commands that may have failed in the production testing machines.

If you want to contribute to mozharness, the code is at our mercurial repository. If you want to find out more about mozharness, drop by our #releng IRC channel on the mozilla irc server. (irc.mozilla.org)

I Compiled Servo and Here’s What It Looks Like

Screen Shot 2013-07-01 at 11.52.34 PMIt’s nothing fancy, I know, but it’s pretty impressive. It can render some CSS already. It’s quite a feat considering that it was announced only 2 months ago and it’s built using rust, a new programming language from Mozilla.

In case you’re unaware, servo is an experimental web browser engine developed by Mozilla and Samsung.

If you want to play with the browser or contribute to it, follow these steps to compile it on your own machine.

Compiling Servo

First, clone your own copy of the source.

$ git clone https://github.com/mozilla/servo.git

Next, create a build directory and compile it with make.

$ mkdir build && cd build && ../configure && make

And you’re set. Once it’s compiled, you should find a servo executable in the build directory. Enjoy and happy hacking.

Collection of FP Articles by Paul Callaghan

Thinking Functionally with Haskell

Dependent Types

OCaml, ocamlfind and camlp4 quirks

I’ve been playing around with OCaml for my final year project. The language is awesome but the ecosystem seems lacking in many ways. Compared to other languages like Ruby, Python and even Haskell, OCaml doesn’t seem to have a lot of people using and talking about it.

The biggest problem I find when using ocaml is the build system. There are a lot of tools to simplify the build process. There’s ocamlbuild and ocamlfind. But I find that the documentations aren’t very helpful. For example, I can’t find any documentations about writing META files for ocamlfind. Neither can I find any guides about writing _tag files for ocamlbuild. I guess the best way around this is for the users of these tools to post their own guides and document their issues. Sp I’m going to write this post so that I can save another user the headaches that I’ve went through.

For my final year project, I’m working with Jeremy Yallop’s deriving library. The library uses camlp4 (original syntax). To install the library to your system, install opam and execute this command:

$> opam install deriving-ocsigen

Opam is a package manager for OCaml libraries. After installing, make sure you have findlib installed. Then test your installation:

$> ocaml
        OCaml version 4.00.1
# #use "topfind";;

You should see a menu with some commands to get you started with findlib. Next execute these commands in your ocaml toploop:

# #camlp4o;;
# #require "deriving-ocsigen.syntax";;

You should see some output showing that the library’s path is added to ocaml’s search path. Next, let’s try to evaluate some code that use the deriving library.

type point = { mutable x : int; mutable y : int }
     deriving (Eq, Show);;

If the above code is successfully evaluated, then you should have the deriving library installed on your system. To compile code that uses the deriving library, you have to use ocamlfind telling it to use the camlp4o (original syntax) and to use the deriving-ocsigen.syntax package.

ocamlfind ocamlc -package deriving-ocsigen.syntax -syntax camlp4o -c point_seq.ml -o point

That’s it. Hope I’ve saved you some headaches when using ocamlfind with libraries that use camlp4. I believe the above commands should work with other libraries that use camlp4 (sexplib) too.

Python Closures and the Python 2.7 nonlocal Solution

In python, functions are first class. A language with first class functions allows functions to be passed as a variable and returned from a function just like other first class data types like integers or floats. Some notable language with first class functions are Python, JavaScript, Scheme and Haskell.

Therefore in python, it’s possible to do things like this:

def raise_to_base(n):
  def my_pow(x):
    return x**n
  return my_pow

>>> pow_base_5 = raise_to_base(5)
>>> pow_base_5(2) # => 32

(above code is taken from Python Closures Explained)

I have created a function that returns another function. And the inner function is known as a closure. A “closure” is an expression (typically a function) that can have free variables together with an environment that binds those variables (that “closes” the expression). With closures, we can do other neat things like this:

def outer():
  y = 0
  def inner():
    nonlocal y
    y+=1
    return y
  return inner


>>> f = outer()

>>> f()
1
>>> f()
2
>>> f()
3

The inner function now becomes something like a method in OOP. It controls the way the variable y is accessed from the outside. But there’s a problem when we want to do this in Python 2.7. The above code works only in Python 3.x. The nonlocal keyword does not exist in Python 2.7. To solve this problem, we can use dictionaries (or we can also create another object) to store the y variable in a namespace that the inner function can access.

def outer():
  d = {'y' : 0}
  def inner():
    d['y'] += 1
    return d['y']
  return inner

Though IMO, it’s kinda weird that we can’t access the non local variable but we can access the dictionary.

Connecting to NUS wifi on Arch

I recently have to move to arch due to some annoying gcc problems on the OSX of my macbook but it seems that configuring the wifi to connect to the NUS network on arch isn’t that easy.

The NUS wifi network is configured with WPA2 enterprise authentication. I referred to Tech That!’s blog post to configure it.


First, you have to get the netcfg and the wpa_supplicant packages.

sudo pacman -S netcfg wpa_supplicant

netcfg allows you to configure your wifi according to profiles and manages the profiles for you. wpa_supplicant is used for configuring the authentication.
Next, copy /etc/network.d/examples/wireless-wpa-config to /etc/network.d/nus. In this example, I shall name it as nus.

sudo cp /etc/network.d/examples/wireless-wpa-config /etc/network.d/nus

Then, make sure your /etc/wpa_supplicant.conf looks like this:

ctrl_interface=/var/run/wpa_supplicant

network={
ssid="NUS"
key_mgmt=WPA-EAP
eap=PEAP
group=CCMP
pairwise=CCMP
identity="NUSSTU\"
password=""
priority=1
ca_path="/etc/ssl/certs"
phase2="auth=MSCHAPV2"
}

Lastly, start the netcfg profile. The profile name should correspond to the file name of the profile configuration file in /etc/network.d/.

sudo netcfg nus

Javascript Shell in Your Terminal

I’m currently working on the Sage Notebook and I’ve been writing code in javascript. To test snippets of js code, I usually open up either firebug or the dev console on Chrome to test it. Which is pretty annoying. I have to wait for the browser to load, then I’ve to wait for firebug to load, then I have to click on the console prompt in firebug. So, I’ve decided to look for a quicker solution to my problem. I’ve previously seen a talk that a web developer gave (it’s really funny, you should watch it!) and he gave live demos of javascript running on his command line. Kinda like a javascript repl. So I’ve decided to look for it and I found rhino.

Rhino is an open-source implementation of JavaScript written entirely in Java. It is typically embedded into Java applications to provide scripting to end users. It is embedded in J2SE 6 as the default Java scripting engine.

The best thing about rhino is that it has a javascript shell. (IMO, that’s the best thing about it. It comes with other cool things like a debugger by the way.)

Setting Up Rhino (On the OSX)

First, download Rhino. Extract it. Then, create the ~/Library/java/Extensions directory if it’s not created yet. Lastly, copy js.jar from the rhino folder to ~/Library/java/Extensions. Or you can copy and paste the following commands to your Terminal.

mkdir -p ~/Library/java/Extensions
cp ~/Downloads/rhino1_7R3/js.jar ~/Library/java/Extensions

To test it, execute this in the Terminal:

java org.mozilla.javascript.tools.shell.Main

You should see this in your Terminal:

Rhino 1.7 release 3 2011 05 09
js>

And..you’re done. Oh a quick tip, instead of typing that long command to your terminal, you can alias it. You can do so by putting this in your ~/.bash_profile:

alias js="java org.mozilla.javascript.tools.shell.Main"

Enjoy!

Note: If you want to have readline capabilities like line editing, history, etc, you might want to put this in the ~/.bash_profile instead:

alias js="rlwrap java org.mozilla.javascript.tools.shell.Main"