Brtdku

(Question in the end)
I have a table like this one:

   Date          Period  ID Multi    Volume Price 

 16/01/2019        1    1 -5124.00   -300  34.16    

 16/01/2019        2    1 -5124.00   -300  34.16   

 16/01/2019        2    2  -993.30    -60  33.11    

 16/01/2019        2    3 -4783.00   -200  47.83    

 16/01/2019        2    4  5906.25    150  78.75   

 16/01/2019        2    5 -3342.50   -382  17.50   

 16/01/2019        2    6  8745.00    220  79.50  

 16/01/2019        2    7 16050.00    300 107.00    

 16/01/2019        3    1 -5124.00   -300  34.16  

 16/01/2019        3    2  -993.30    -60  33.11  

 16/01/2019        3    3 -3955.00   -200  39.55   

Period is hours. When new data is published can both contain new data for example Hour 4 ID 1 but also for hour 2 and then the new data recives the ID 8 for Hour 2.

The new data would then look like this. It does no longer show hour one because it belongs to the past. it shows a new observation for hour2 hour5, no for H4.

   Date          Period  ID Multi    Volume Price   

 16/01/2019        2    1 -5124.00   -300  34.16   

 16/01/2019        2    2  -993.30    -60  33.11    

 16/01/2019        2    3 -4783.00   -200  47.83    

 16/01/2019        2    4  5906.25    150  78.75   

 16/01/2019        2    5 -3342.50   -382  17.50   

 16/01/2019        2    6  8745.00    220  79.50  

 16/01/2019        2    7 16050.00    300 107.00 

 15/01/2019        2    8  16050.00    300 107.00   

 16/01/2019        3    1 -5124.00   -300  34.16  

 16/01/2019        3    2  -993.30    -60  33.11  

 16/01/2019        3    3 -3955.00   -200  39.55  

 16/01/2019        5    1 -3955.00   -200  39.55     

How do I merge the new data with the old.
Extra info is that I on some days dont recive any new info.

The ID is trades for that hour. Every new trade for a specific hours gets an new trade ID. I get the data every time there is (a) new trade(s). Trades can happen 24 hours in advance.. When a new trade has happend I recive the CSV file with all confirmed trades from now to +24H and not only the new trade. LIke the table i showed. So I have to finde the difference between the old dataset and the new csv file and then add that new row/rows to the old dataset and wait until I receive a new csv file

I just want to open the old dataset and write something like

 if Date_new = Date_old Then

     If Period_new = period_old Then

        IF ID_new > ID_old Then

            Add the new rows to old dataset.

        end if

    else if Perioed_new > period_old Then

        Add the new rows to old dataset

    end if

 else if Date New > Date_old then

      Add the new rows to old dataset

 end if



sort dataset by date -> period -> ID.

save dataset.     

Im transition to R from matlab and imstuck with this part.
I can get it to look if there is a csv then open the csv's transfrom the raw data into the structure i want and save it to an database.

Hope somebody can give a few hints on how to model the dateset. I spent Hours looking for at solution to this relatively simple problem...

Best
Frederik

I cannot comment with my low rep so I will delete this soon. The behavior of your new data (how something is assigned an "8" versus something else), and the behavior of your current and new data (e.g., is your data live and is writing new files every X hours, or is it overwriting these files?) What is your end goal (are you trying to import all the files into R to manipulate/analyse)?

Providing a reproducible example will probably help you improve the clarity of your question.

finally...

I love packages but they are making me lazy.

library(gtools)

mydata <- smartbind(data1, data2) ' binds the data together in a clever fashion.

library(tidyverse)

cleanmydata <- mydata %>% distinct() ' cleans rows containing the same values in all columns.

done