An Out-Of-Memory exception during structure learning is most likely caused by learned structure inducing too large conditional probability tables. The size of a conditional probability distribution is exponential in the number of conditioning variables (i.e., parents in the graph). If, for instance, all variables have ten states and a variable has 20 parents, then the size of the conditional probability table for this variable is 10^21, which is much more than a standard 32bits computer can manage.
There are a couple of solutions that may apply depending on your data and domain. The lower the significance level used in the statistical tests, the fewer edges will appear in the resulting graph (for instance, you may consider using 0.01 or 0.001). Another approach is to used a special class of models such as the Naive Bayes Model. This model is well suited for classification tasks.