|
(c) 2006, Howard E. Motteler Assigned Wed 22 Feb 06, Due Wed 8 Mar 06, 60 pts Project GoalsThe purpose of this project is to gain experience programming matrix operations with message passing, with the problem size chosen to emphasize performance issues. We will use matrices with on the order of 10e7 elements for the larger tests.
The ProjectYou are to write a parallel program using the MPI libraries to do a three-layer "forward" neural network calculation. You will read in weight matrices, bias vectors, and a data matrix, and write out the result. An important part of the project is to be able to work with matrices whose cumulative sizes may be larger than the memory available on any single processor.A three layer forward calculation can be represented as
y = W3*t(W2*t(W1*x + b1) + b2) + b3
where x is a column vector input, y a column vector output, W1, W2,
and W3 are matrices, b1, b2, and b3 are vectors, and t() represents
the application of the hyperbolic tangent function to each element
of a matrix or vector. This can be generalized to an operation from
matrices X to matrices Y if we duplicate columns of the bias vectors
so that the additions conform. Your program should read the W matrices and b vectors from one file, the data matrix X from another file, and write the result matrix Y to a third file. The format of the files is chosen to be as simple as possible while still allowing for varying-sized data sets. The file containing the weight and bias data is organized as
m1,n1,W1,b1,m2,n2,W2,b2,m3,n3,W3,b3
where W1 is an m1 by n1 matrix, b1 an m1-vector, and so on. The m-
and n-values are 32-bit ints, while the W and b values are 32-bit
floats. The matrices will conform for the supplied test data, but
you still might want to check for that. The matrices are stored in
COLUMN ORDER. The data is all in native-mode big-endian format, so
you can read it with fread() from the blades. The X and Y matrix
files are similar but simpler. The X matrix file has the format
m0,n0,X
and the Y matrix is similar. The default filenames are "Wset.dat"
for the weight and bias set, "X.dat" for the input matrix, and
"Y.dat" for the output matrix. You can add command line (or
e.g. namelist) options for other filenames if you'd like, but you
don't have to. With no arguments your program should use the
default file names. Your program should have three stages: (1) read the data and distribute it to the processes, (2) do the forward calculation, and (3) gather the results back to an i/o process and write the results to a file. At the end of a run you should print the wall time for stage (2). If you want, you can use stage (1) for an initial transpose as the data is read in, and stage (3) for a transpose as the data is read out, but any calculation or data movement beyond that should be done in stage (2). You can save considerable space in memory and in messages by keeping the arrays and other working data in single precision (32-bit) format. In practice neural nets can be very robust numerically, at least with normalized data, so there's no advantage in using double (64-bit) precision. I will provide several sample test data sets with Wset, X and Y files, in blade native format. Note that due to numeric issues, your Y values may not be exactly the same as in the sample data, but individual values should be very close, say within 10e-6.
Getting StartedA good way to get started is to write a sequential version of the program first. This will resolve basic issues like i/o, managing arrays, and so on. Then choose a parallel design, focusing on how you want to distribute data across the cluster and move it during the course of your computations. Class discussions and Section 11.2 of your text should help with this.
Submitting Your ProjectMake a tar file with everything I will need to build and run your code, including a Makefile, optional README, and any supporting documentation you think is relevant. But don't bundle in any ".doc" files, I don't want to have to fire up Open Office. Make sure that your name and "project 1" are at the top of your main files. Mail this to me as an email attachment, with subject "project 1". Do not mail data sets unless they are relatively small and there is a good reason to do so.
GradingPlease read the notes on general information on programming projects. Half your grade will be based on how well your code works, one quarter on design and code sanity, and one quarter on documentation. The documentation should include an honest evaluation of how well your code works, as best you are able to determine. Projects are due by midnight of the assigned date. There is a 5% bonus for each day a project is turned in early, for up to two days early, and a 5% penalty for each day the project is turned in late. |