Datanomics

Here are some things I learned today about my Mac:

•  dealing with Excel dates in Mac Excel and Windows Excel
• remove annoying applications from “Open With”
• (sort of Mac) Excel dates in Matlab: add a constant!

The new Mac OS Leopard uses a different graphics engine which Matlab 7.4+ does not like.  Working in the UI is painful.  No more!  Just follow this simple solution to bring back smooth, fast scrolling.

The Quantum Random Bit Generator Service has made available a random number generator that uses quantum theory to create real random numbers:

We use ‘Quantum Random Bit Generator’ (QRBG121), which is a fast non-deterministic random bit (number) generator whose randomness relies on intrinsic randomness of the quantum physical process of photonic emission in semiconductors and subsequent detection by photoelectric effect. In this process photons are detected at random, one by one independently of each other. Timing information of detected photons is used to generate random binary digits – bits. The unique feature of this method is that it uses only one photon detector to produce both zeros and ones which results in a very small bias and high immunity to components variation and aging. Furthermore, detection of individual photons is made by a photomultiplier (PMT). Compared to solid state photon detectors the PMT’s have drastically superior signal to noise performance and much lower probability of appearing of afterpulses which could be a source of unwanted correlations.

They have made a C++ and Matlab toolbox available.

If you ever import data into Matlab in tab or comma delimited formats, be warned:

 CSVREAD fills empty delimited fields with zero.  Data files where the lines end with a comma will produce a result with an extra last column filled with zeros.

“dlmread” also suffers from this problem.  So if you want to import a data file that possibly has null values, make sure the original source has some (usually numeric) way of distinguishing them.  Or, you can use ‘textscan‘ which replaces “null” with “NaN.”  The function requires more manual inputs, but unlike ‘csvread’ or ‘dlmread’ it can be adjusted for almost any quirky data.

If you have an FTP server set-up somewhere with ample space and bandwidth, Matlab can store and retrieve its files and data remotely. Just use these simple commands to connect and disconnect:

% File to connect to the betanaught.com server where my Matlab/data resides

% connect to the db
f = ftp('yourdomain.com', 'user', 'password');

% change the directory
cd(f, 'matlab');

% now change the directory that we want to download the directory
cd '/';

% Download the directory
mget(f, 'remote_dir');

% *****************
% INSERT PROGRAM HERE
% *****************

% when done move a directory here
cd ..
% now move the directory back to the server
mput(f, 'remote_dir');
disp('Files have been put back on the server');

% close the connect
close(f);

Let x be your vector of data which has to be bootstrapped. For each instance of the loop write:

x = x(floor(rows(x)*rand(rows(x),1))+1,:); % bootstrap observations

Now we have a matrix with the same number of rows and columns, but with re-sampled data with replacement. A one line bootstrap! (Modified code from John Cochrane)

If a Matlab script generates a generic diary file, but differs based on the parameters you send the script, you may want to introduce a renaming procedure for your diary files. Suppose that script1.m calls script2.m and passes it some parameters. The latter script is written to produce a diary file ‘run.txt.’ We want to distinguish the results of the parameter settings in script1.m by giving it its own diary name ‘parameters1.txt’ which you assign to variable diary_name. To accomplish this, simply add the following code to script2.m before you turn the diary off:

movefile('run.txt', diary_name);

Note that you could also you this functionality to name your diary files based on time stamps by simply assigning diary_name like this:

diary_name = datestr(now);

Matlab’s disp command display text or variables on the screen and prints it to the diary. Unfortunately, it does not allow one to combine the two. Enter fprintf. Suppose we want to print to the screen the number of remaining observations. With disp this is done as follows:

disp('Observations left:');
disp(num_observations);

disp automatically generates a new line character (\n) after each command, so this looks like:

Observations:
37000

fprintf is both more intuitive and easier on the eyes. To again display the number of observations we write:

fprintf('There are %g observations left in the data.\n', num_observations);

Here, ‘%g’ is replaced with the variable after the comma, “num_observations” and prints:

There are 37000 observations left in the data.

There are many options for the %g that define the conversion. I have found %g to be satisfactory. Check the fprintf page for all the options or type help fprint at the command prompt.

The rand function in Matlab generates an uniformly distributed random number or set of numbers. If we want to generate a random integer between 1 and n, we type:

floor(rand(1,1)*n) + 1

This creates a random number between 0 and 1 (inclusively) and multiplies it by the maximum integer we are interested in. Then it rounds down so we have an integer and adds one so that the final result is between 1 and n.