Doc version: 1.o, Plugin: 30014
- Bi-directional websocket communication
- Support to backfill historical data
- Easy to integrate Json message structure
- Run multiple instances of AmiBroker locally, each connected to one of multiple Servers
- The design makes it Broker / Data vendor agnostic
- Runs a python based relay server, to ensure individual connections don't disrupt your flow
- Build your own data routing logic
- Store your time-series data in the superfast Amibroker storage database
- Get you Realtime Quote window ticking as opposed to plain ASCII imports
- ArcticDB active integration
- A lot of leverage with keeping bloated data locally in ArcticDB and a concise Amibroker database
============================================
WS_RTD requires AmiBroker to run.
| Software | Supported / Tested version |
|---|---|
| AmiBroker | v6.30+ x64-bit |
| Visual C++ | VC++ 2022 redistributable |
| Windows OS | Windows 10/11 x64-bit |
| Linux OS | To-Do |
| Vanilla Python | 3,12 x64-bit |
| Install the above dependencies. | |
| Sample codes like sample_server.py requires additional python packages to be installed. |
Copy WsRTD.dll to C:\PATH\AmiBroker\Plugins folderhttps://www.amibroker.com/guide/w_dbsettings.html
File > New > Databaseselect Database folder
Datasource: WsRtd data Plug-in
Local data storage: ENABLE (very important)
set: Number of bars
Base time interval: 1 minute or as suitedTo-Do: Enabling seconds(timeframe) in next iteration. Tick-data unsupported for now
[
]
Click Configure button[
]
The internal plug-in DB stores RTD Quotes as a RING BUFFER. Max_Num_Sym_Quotes here means the number of bars to hold. If you run Analysis frequently and ensure all symbols are accessed, then you can set it as low as 10 to reduce memory usage. Each bar is 40 KB.
- Max_Num_SYM_Quotes : Requires AB restart
- *Changes in IP / Port / AUTH : Requires Plugin shutdown / connect, ie. Reconnect.
ensure these Python packages are installed and their dependencies
import asyncio
import websockets
import datetime
import json
import random
import sys
import pandas as pd
import copystart the sample server in a Windows terminal,
cd \path\to\server
py sample_Server.py[
]
[
]
When Data Plug-in connects to the server
[
There are 4 types: Json-RTD, Json-HIST, Json-CMD, Json-ACK
Currently, minimum periodicty/timeframe/interval of bars is 1 minute. Valid fields in the json object ORDER See the Json recommended section, rtd message has to start with "n" field. Subsequent order does not matter. STRUCTURE This is Array of JSON strings
'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 'x1', 'x2', ( for Quotations ) 's','pc','bs','bp','as','ap', 'do', 'dh', 'dl' ( for Realtime Quote window)
'n'=symbol name, date, time, open, high, low, close, volume, open interest, 'x1'=Aux1, 'x2'=Aux2 's'=total daily vol, 'pc'=prev day close, 'do', 'dh', 'dl', are Day Open, High, Low 'bs', 'bp', 'as', 'ap', are bid size/ bid price /ask size / ask price
Sample rtd json
[{"n":"SYM1","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1},{"n":"SYM2","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1},{"n":"SYM3","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1}]Supported Date/Time formats:
milliseconds not yet supprted
u = unix timestamp localtime() ( type: integer64 )
g = unix timestamp gmtime() ( type: integer64 )
d = date ( type: integer )
t = time ( type: integer )
u, g and d are MUTUALLY EXCLUSIVE, but any one is mandatory.
"t" is mandatory for RTD along with "d"
/U=unix_local_ms /G=unix_utc_ms (To-Do:)Refer to Amibroker ADK link above for data types. 'd'.'t','as','bs' as integer, float is safe for the rest. To-do: unix timestamp(u/g), seconds interval
if required fields are not found, they are set to ZERO & that will distort candles.
o, h, l, c = open,high,low,last_traded_price = mandatory for new Quote
v, oi, x1, x2 = optional and set to 0 if absent
Realtime Quote window variables are optional, but they
are required for this feature to work properly.Using d,t is more efficient over u,g as 1 step less in conversion and should be preferred if available.
Parse a history data JSON string that is used to backfill a Ticker
This is all the backfill records in Array format for a single ticker format specifier is fieldsStr = "dtohlcvixy" only valid fields should be set and in that particular order.
ORDER The sequence of columns in hostorical data can be anything set by the user, but it has to match its character in the Format specifier string. The beginning of the json message will start with "hist" containing Symbol name followed by "format" field containing the Format specifier string.
Important: History Array records must be in ascending order, ie. Least DateTime first.
(ugdt)ohlcvixy
milliseconds not yet supprted
u = unix timestamp localtime() ( type: integer64 )
g = unix timestamp gmtime() ( type: integer64 )
d = date ( type: integer )
t = time ( type: integer )
u, g and d are MUTUALLY EXCLUSIVE, but any one is mandatory.
"t" is mandatory for intraday bars with "d"
o = open price ( type: float ), required
h = high price ( type: float ), required
l = low price ( type: float ), required
c = close price ( type: float ), required
v = volume ( type: float ), optional
i = open interest ( type: float ), optional
x = aux1 ( type: float ), optional
y = aux2 ( type: float ), optional
/U=unix_local_ms /G=unix_utc_ms (To-Do:) if required fields are not found, they are set to ZERO & that will distort candles.
Sample Date-Time DATE (optional TIME) d,t is supplied as INT,
{"hist":"SYM1","format":"dtohlcvi","bars":[[20240601,100000,22.1,22.5,22.1,22.8,120,0],
[20240602,110000,22.8,22.7,22.3,22.4,180,0],[20240603,120000,22.8,22.7,22.3,22.4,180,0]]}Sample unix timestamp
({"hist":"SYM1","format":"uohlcv","bars":[[1731670819,22.1,22.5,22.1,22.8,120],
[1731761419,22.8,22.7,22.3,22.4,180]]}u=Unix Local timestamp OR g=Unix GMT/UTC timestamp that is use ( windows struct tm = localtime() or tm = gmtime()
Currently, Vendor backfill means N most recent days data so plugin also sets Bars from the first of this Array to the last bar the existing bars are overwritten, from first bar of hist. old_data + WS_data Now, backfill means old_data + backfill (upto last backfill) then, old_data + backfill_data + Rt_data ( from bars after last backfill bar, if any )
[
Inserting historical data into AmiBroker Database
Work in progress, subject to change.
ALL REQUESTS made by plug-in TO Server have only "cmd" and "arg fields" Server should REPLY to all above REQUESTS with same "cmd" and "arg" fields. Additionally inserting the "code" which is integer. Code values for OK=200 to 299, and OTHER/BAD=400 to 499. Specific codes are not defined yet.
ALL REQUESTS made by Server To Plug-in will have valid "cmd" and "arg" fields. Additionally, all SERVER-side requests will have code=300 which is integer. ( The only exceptions is json-RTD and json-HIST formats.)
Plug-in will respond with Acknowledgement TO Server for all code=300 Server-side REQUESTS with json-ACK "ack" with server-request, "arg" with description and "code" which is integer Code values for OK=200 to 299, and OTHER/BAD=400 to 499. Specific codes are not interpreted currently.
This is a special automatic backfill request command that is sent to the server when a symbol is accessed in Amibroker for the first time. Server can choose to ignore this request or respond with a json-History backfill data of a desired period. {"cmd":"bfauto","arg":"SYMBOL_NAME DateNum TimeNum"} last Quote timestamp sent when symbol has some data {"cmd":"bffull","arg":"SYMBOL_NAME"} sent when symbol has NO data
{"cmd":"bfauto","arg":"SYM3 20241125 134500"}
{"cmd":"bffull","arg":"SYM9"}{"cmd":"bfsym","arg":"reserved SYMBOL_NAME int_preset"}
{"cmd":"bfsym","arg":"y SYM1 3"}{"cmd":"bfall","arg":"x"}
{"cmd":"bfall","arg":"x"}For ALL Symbol backfill, client application should still send individual Json-hist messages for each symbol.
{"cmd":"addsym","arg":"SYMBOL_NAME"}
{"cmd":"addsym","arg":"SYM10"}{"cmd":"remsym","arg":"SYMBOL_NAME"}
{"cmd":"remsym","arg":"SYM6"}{"cmd":"cping","arg":""}
{"cmd":"cping","arg":""}Get state if any Client-App is connected to the relay. Also, a ping-alive packet that wsrtd is connected to Relay server.
{"cmd":"CMD_SENT","code":int_code,"arg":"response string"} Mandatory code field
{"cmd":"remsym","code":200,"arg":"SYM6 unsubscribed ok"} /* sucess example*/
{"cmd":"addsym","code":400,"arg":"XYZ9 subscribe error, invalid"} /* failure example*/Mandatory code field
{"cmd":"cping","code":200,"arg":"Vendor Connected"} /* Client running & connected to remote*/
{"cmd":"cping","code":400,"arg":"Vendor Disconnected"} /* Client running but remote source is disconnected*/Mandatory code=300 {"cmd":"request_cmd","code":300,"arg":"request_arg"}
{"cmd":"dbremsym","code":300,"arg":"SYMBOL_NAME"} Case-sensitive match This is useful to FREE memory in the plug-in DB
{"cmd":"dbremsym","code":300,"arg":"SYM9"}
returns {"ack","code":200."arg":"SYM9 removed from DB"} /* success */
returns {"ack","code":400,"arg":"SYM9 NOT found in DB") /* failure */{"cmd":"dbgetlist","code":300,"arg":""} "arg" field required, can set empty or some string description returns a comma-string of symbols
{"cmd":"dbgetlist","code":300,"arg":"DB Symbol list requested at 14:56:00"}
returns {"ack":"dbgetlist","code":200,"arg"="AA,AAPL,AXP,BA,C,CAT,IBM,INTC,"}{"cmd":"dbgetbase","code":300,"arg":""} "arg" field required, can set empty returns a STRING with base time interval in seconds(convert to INT)
{"cmd":"dbgetbase","code":300,"arg":"DB Symbol list requested at 14:56:00"}
returns {"ack":"dbgetbase","code":200,"arg"="60"}{"cmd":"dbstatus","code":300,"arg":""} "arg" field required, can set empty returns a STRING with Format as ( convert to INT ) Current_Sym_Count, Max_Sym_Limit, Max_Size_Sym_Quotes, Refresh_Interval, Base_Interval, DB_Name
{"cmd":"dbstatus","code":300,"arg":"DB Status requested at 14:56:00"}
returns {"ack":"dbstatus","code":200,"arg"="500 1000 200 300 60 Db_Test"}Plug-in will respond with Acknowledgement TO Server for all code=300 Server-side REQUESTS with json-ACK "ack" with server-request, "arg" with description and "code" which is integer Code values for OK=200, and OTHER/BAD=400
For samples, check 3.3 Plug-in return messages- Json message should be compressed removing all whitespaces or prettify.
- Use your library Json Encoder to prevent errors. For example, The JSON standard requires double quotes and will not accept single quotes
- The json-type of messages are string matched at the start
- Json CMD, history-format string and RTD fields are all case-sensitive
C++ case-sensitive Raw string match example for performance
R"([{"n")" // Realtime Data
R"({"cmd")" // Commands
R"({"hist")" // Historical DataExplained here Amibroker Forum Amibroker Batch window manual
Every time the Plug-in receives data, it stores is internally in its own temporary storage and notifies Amibroker of new data. Unfortunately, by design, if the Symbol is not being used by the user in any way, the plug-in may never be able to "push" the data to AB local Database that is persistent. Therefore, user needs to use a Batch file that consists of an empty Analysis Exploration to force all symbols to fetch data. More on this later.
These are some ways to communicate with the Plug-in from the Amibroker User-Interface OR when the websocket is shutdown and therefore not reachable externally.
Batch command to inform plug-in that an all symbol fetch has been performed.
Batch command to start the Websocket connection.
Batch command to stop the Websocket connection.
Batch command to log some Plug-in statistics.
[1] DBG: Db() Status Map_size=4, Qt_RingBufr=200, SymList=SYM1,SYM3,SYM4,SYM5,For every first access of a symbol, bfauto or bffull request is sent once only. You can clear the flag with this.
< Add more commands as required>
WS_RTD uses extensive logging using Windows Debug output.
- Just run DebugView
- Set the Filter to "DBG" (may change this string later)
- View realtime Plug-in related information here
[
]
Check under Symbol > Information window,
Use only local database = No (Ensure No, If yes, plug-in-cannot update it)Whenever new data arrives, whether RTD or Historical, it is immediately stored internally. Amibroker is notified, but AB will choose to to request data only when that symbol is being used. To visualize that, currently, Two fields in the AB REaltime Quote window are dedicated to that.
EPS: This column shows the Total count of bars in Plug-in DBDividend: This column shows the Total count of bars in updated to AB from the Plug-in DB[
]
These commands will only Connect or Disconnect the plug-in (Client Websocket) to the local relay server. Data stored in the Plug-in DB is not affected. AB Symbols will still get updated if there is fresh data in the plug-in DB while the user has disconnected the plug-in. The plug-in DB is not persistent, that means, if user Exits Amibroker or Changes the Current DB in Amibroker, only then the plug-in data is cleared.
DBG: ConfDlg() RI=300ms, Conn=127.0.0.1:10101, SymLimit=1000, QuoteLimit=250, DbName=Data< more items to come >
The plugin stores settings in the Windows Registry like Amibroker QT.dll Amibroker path
Computer\HKEY_CURRENT_USER\SOFTWARE\TJP\wsRtDNested in TJP\wsRtD\<Database_name>Settings will be UNIQUE to each unique database name created in AB. ALL Amibroker Databases that share common Database name but are different filesystem path will "still" share the same settings. This is the best of both, allows the user to separate or share settings as required.
Example 1:
Amibroker DB name: Data with their respective paths
C:\sample1\Data
D:\archive\old_sample\DataBoth AB DBs above will share the same settings and there will be only one registry key named "Data"
Example 2:
C:\sample1\Data_rtd
C:\sample1\wsrtdData2 and wsrtd will be a new entries and each can have its own settings Image of Windows Registry below:
[
]
For production release:
To do:
For now, this integration depends on the user and technical skill
Client-sender program should be developed using Vendor specific library.
Working on providing a Relay ServerNote:
--rolesendis required for client-sender identification
Example rolesend: data vendor / broker client program / Python sample server
Note:
--rolerecvis required for client-receiver identification
Example rolerecv: Ws_Rtd plugin or ArcticDB data collection socket conn
Verify the connection by navigating to your terminal running the server
127.0.0.1:10101The design itself keeps in mind a minimal requirement so the application just needs to conform with the fol1owing:
- Client-Sender connects to the matching websocket port
- On connection, it sends the above simple string "rolesend" before any other data
- Then it follows the json-RTD format for realtime bars matching the interval set in AB configure DB
- To handle Requests from Data Plug-in, it will receive the json-CMD or json-Hist. It should respond with the appropriate json formats described above. That's it.
- The Advantage of using a relay server is that when either Plug-ins or Client applications reconnect or disconnect, they don't cause the other party websocket drop errors which is distracting as well.
< more documentation on this later > The Programming Language of choice or platform does not in any way become restrictive to the user.
As the websocket is disconnected, no commands can be sent or received, therefore it is not possible. However, Data available inside the plugin-DB can still be accessed for symbols to get updated.
By design, it is up to the user to ensure that duplicate data is not imported. Mixing two different methods can cause unwanted behaviour. You can delete all the data and import it afresh. Some steps are taken when using plug-in to overwrite duplicate data. However, this Plug-in is quite technical and for performance, the user's Client side application should ensure data integrity.
3) Some of my Symbols are not updating with data but my Server shows that data for RTD is generated.
Ensure that under Symbol Information window in Amibroker, the setting "Use local database only" is set to No. If yes, Plug-in cannot update to it.
When new symbols arrrive in the plugin-DB, the plugin status will be dark Green in colour. You can go to Configure settings Dialog and Click RETRIEVE Button. Symbols will automatically get added.
[
]
The status color will change to 🟢 from
Kindly use DebugView to check if settings change requires an AB to be restarted or Plugin to be reconnect. Scroll up to Configure section and read the details.
https://www.amibroker.com/guide/afl/getextradata.html
if Data plugin websocket is connected, 1 else 0
if Client App websocket is connected, Client App should implement "cping" cmd
0 = client-app not running
200 = running
400 = running but remote disconnected
-1 = ping sent, awaiting replyWant to contribute? Great!
For now Clent-sender applications have a lot of scope for expansion Current Relay Server is python-based. Client-sender can be in a Programming Language of your choice or using Data Vendor library, these applications can be expanded greatly to encompass various Brokers APIs and Data Vendors.
WS_RTD uses C++ for development, strongly based on ATL/MFC, but is currently closed.
To be decided
This LICENSE Section alongwith the CREDIT and AUTHOR Section at minimum should be distributed EXACTLY as-is when distributing a copy of the data-plugin binary or DLL.
NSM51 A fan and avid Amibroker user. https://forum.amibroker.com/u/nsm51/summary < Journey story about the idea of this plugin >
Inspired by QT.dll sample from AmiBroker ADK
https://gitlab.com/amibroker/adk/-/tree/master/Samples/QT
Credits: Amibroker Company & Dr. Tomasz Janeczko
Current WS library: EasyWsClient
https://github.com/dhbaird/easywsclient
License: MIT
Works well as light-weight windows WinSock2 wrapper.
Credits: dhbaird
Current JSON library: RapidJson
https://github.com/Tencent/rapidjson
License: MIT
Credits: Milo Yip
RapidJson is benchmarked as fastest.
https://stackoverflow.com/a/69054531, Credits: Jerry Coffin
Created using Dillinger