Work in R with very large data set -


I am working with a very large data set which I am downloading from Oracle data base. There are approximately 21 million rows and 15 columns in the data frame. My OS is Windows XP (32-bit), I have 2 GB RAM. I can not upgrade my RAM or my OS (it's working before I take a decent PC before it will take months before ). 

  Library (RODBC) sqlQuery (Channel1, "Select * from table 1", stringsfactor = falls)   
 I already have "vector from here" Can not allocate XMB to "I got stuck with. I got a suggestion about using the  ff  package. I would like to know if someone familiar with the FF package can tell me that this will help in my case. Do you know another way to get around the memory problem? Will 64-bit solution help? Thanks for your suggestions.   
 
 
 If you are working with package FF and have your data in SQL, you can easily Bring them in the FF package, see the documentation for example when using ROracle  
 In my experience, the FF is fully compatible with the dataset you are working with (21 meo rows and 15 columns) Friendly - in fact your setup is smaller up to FF, unless your column has a lot of variables Fitter data that will be converted into factors (meaning that all your factor levels should be able to fit in your RAM) Package ETLUtils, ff and package to get your data in RF and some basic on the basis of FF Do statistics. Depending on what you will do with your data, hardware, you may have to consider sampling while making the model. I am liking my data in R, building a model based on a sample and FF ( Like chunking) or want to create scores using the tool from the package FFbbase.  
 The drawback is that you have to use for the fact that your data are FFDF objects and this may take some time - especially if you are new to R.

Comments

Post a Comment

Popular posts from this blog

Python SQLAlchemy：AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'schema' -

I tried to create a new database in my project, but when I run the script, I get this error, I have another project, this type of definition worked before, but now it gets a single error. I am using Python 2.7.8 and the version of SQLAlchemy module is 0.9.8. By the way, using a project flask- SQLAlchemy, it works well, I'm confused. Traceback information is as follows: Traceback (most recent call final): File "D: / Projects / IOM / db_create.py", line 4, & lt; Module & gt; From the models the base file "D: \ Projects \ OO-IM \ models.py", in line 15, in the & lt; Module & gt; Column ( 'followed_id', int (), ForeignKey ( 'user.id')) file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ schema.py", line 369, __new__ schema = metadata. Schema file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ elements.py", line 662, AttributeError in __getattr__ key): Neither 'column' object nor ...
Read more

java - How not to audit a join table and related entities using Hibernate Envers? -

I use Eners to be hibernate to audit my institutions. I have an audit entity, Foo , which includes a list & lt; Bars & gt; However, I do not want to audit the bar institutions, in the form of properties, I have written that: @inti @edicated public class Foo {@JoinTable (name = "T_FOO_BAR", joinColumns = @JoinColumn (name = "FOO_ID"), inverseJoinColumns = @JoinColumn (Name = "BAR_ID")) @ManyToMany (Cascade = Persist) @ Indited (targetAuditMode = RelationTargetAuditMode.NOT_AUDITED ) Public listing & lt; Bars & gt; GetBars () {Return bars; }} Now, I want to retrieve an amendment of foo : AuditReader reader = AuditReaderFactory.get (GetEntityManager ()); FU modification = (FU) reader.CreativeTeA () .entensEtrevision (Foo Class, 42) .getSingleResult (); Unfortunately, when I want to retrieve all the data (i.e. when it loads lazy times ), error me ORA-00942: table or view is not present , because Tried to query this: Sel...
Read more

mongodb - CakePHP paginator ignoring order, but only for certain values -

While pagging my user model in kppp, I can sort with some values, but not others. For example, I can order results from created or email , for example, but username or reputation seems to come back order by arbitrary command for example with a list of users: $ this-> paginate = array ('condition' => array ( 'User.is_active' = & gt; true), 'border' => 24, 'order' => array ('User.created' => DESC)); Works as expected, but $ this - & gt; Paged = array ('conditions' = & gt; array (' user.isIactive '= & gt; true),' border '= & gt; 24,' command '= & gt; array (' user repeats' = ' & Gt; DESC ')); No, no. I first thought that this could be a database issue, but when I search the database directly, sorted as expected. Note: I am using Mangodebi with the Mongo DB plugin of Ichikawa for the KPPHP; Pagnetizing users are used to work properl...
Read more