Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

6 feat add snap dem manager #10

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
eeef5de
enh: add instruction on how to build the docker image + small fix
Jul 19, 2024
bdc6c4d
Merge branch '1_main_class_and_input_parsing' of github.com:sen2vm/se…
Jul 19, 2024
23ab565
merge 1_main_class_and_input_parsing
marinebcht Jul 29, 2024
e0a512f
merge origin/add_Dockerfile
marinebcht Jul 29, 2024
12362ba
enh: add jaxb dependency + fix run main with java command
Jul 29, 2024
7f268eb
enh: check that detectors/bands array provide by json file respect a …
Jul 29, 2024
daa9a15
feat: add a sen2VMException class to manage all exception throw by se…
Jul 29, 2024
350ef5c
enh: add classes to manage all the constants use in sen2vm-core
Jul 29, 2024
df26600
add GIPP management + update data test
Jul 29, 2024
747bc1b
fix: indentation in pom.xml + clean
Jul 29, 2024
a2b8814
add constantes files
marinebcht Jul 29, 2024
9ce9f76
add images index + grid direct loc chain
marinebcht Jul 29, 2024
9f6fd98
mod: docker / file
marinebcht Jul 29, 2024
fdde789
fix: update readme since we fix the bug when running jar with java co…
Jul 29, 2024
eeac71c
merge origin/feat_read_GIPP
marinebcht Jul 30, 2024
3f38db7
merge origin/feat_read_GIPP
marinebcht Jul 30, 2024
b11abbd
FIX sens for band/detector + ADD input parameters maj from input + AD…
marinebcht Jul 31, 2024
362cb4c
refac: handle xml and dbl extensions + raise error when duplicated gi…
Aug 5, 2024
9a91f70
remove useless unionBand object
Aug 5, 2024
aee083a
fix: respect the colon rule of use in english
Aug 5, 2024
c333722
fix: add space after colon
Aug 5, 2024
6365dea
refac: move constants in a same class + remove unused ones
Aug 6, 2024
f480f96
working direct loc grid + mini grid
marinebcht Aug 14, 2024
62649dc
enh: instantiate sxgeo simpleLocEngine and run first location test
Aug 19, 2024
79b0021
add InverseLocGrid class
marinebcht Aug 19, 2024
a410a88
merge with feat_read_GIPP
marinebcht Aug 19, 2024
7579e4c
feat: add datastrip manager
Aug 19, 2024
4d7fb5d
feat: add xsd for datastrip
Aug 19, 2024
b0ac1d4
merge with feat_read_GIPP
marinebcht Aug 19, 2024
69ef415
starting add granule mtd read
marinebcht Sep 2, 2024
8590dd3
commit before merge without static synchro
marinebcht Sep 3, 2024
45b3960
merge without static synchro
marinebcht Sep 3, 2024
8cf8e65
read mtd ds ok
marinebcht Sep 4, 2024
687397e
tif OK
marinebcht Sep 6, 2024
47ba051
Merge remote-tracking branch 'origin/4_feat_add_refining' into 3_read…
marinebcht Sep 6, 2024
3215902
upgrade classes + vrt tricherie
marinebcht Sep 9, 2024
36e8660
my config
marinebcht Sep 9, 2024
88939c7
Merge branch '3_read_mtd_granule_add_other_info' into 3_inverse_loc
marinebcht Sep 9, 2024
ccd6c6c
computeInverseLoc missing
marinebcht Sep 9, 2024
66d0eea
correction du sensor
marinebcht Sep 9, 2024
981931b
feat: when IERS bulletin not providing, use IERS information from dat…
Sep 25, 2024
c945dc9
fix: forget to push utils for previous commit
Sep 26, 2024
93b859f
clean remove useless methods from utils
Sep 26, 2024
1cac454
remove useless import
Sep 26, 2024
4358011
fix: get the best fitted ITRFVersion from the date of input data
Sep 27, 2024
00e965b
Merge branch '5_feat_read_IERS_from_mtd' of github.com:sen2vm/sen2vm-…
Sep 30, 2024
cd8688a
feat: add SnapDemManager to manage snap dem
Sep 30, 2024
7c9a953
use snapdemmanager from main
Sep 30, 2024
76eb38b
direct loc fonctionnel en 2D, à valider ave Jonathan (equation EVEN c…
marinebcht Oct 11, 2024
8f28473
merge direct loc
marinebcht Oct 11, 2024
94f1fce
wip
marinebcht Oct 14, 2024
3aaf9d9
correction jonathan
marinebcht Oct 14, 2024
04ed53d
wip sortie invers loc ok (sans dem)
marinebcht Oct 15, 2024
25085c2
gestion de l'altitude ou non
marinebcht Oct 15, 2024
aa259b0
log erreur hors du modele
marinebcht Oct 15, 2024
358b15d
change default value exportAlt
marinebcht Oct 15, 2024
18e9b7c
wip epsg add to tiff
marinebcht Oct 16, 2024
e6302f7
log + simplification
marinebcht Oct 16, 2024
3a4eb16
new param in config file
marinebcht Oct 16, 2024
9d105ab
merge
marinebcht Oct 16, 2024
6b9344e
if direct/inverse + correction x/y
marinebcht Oct 17, 2024
5646818
correction startX
marinebcht Oct 17, 2024
e30789d
default exportAlt = False
marinebcht Oct 17, 2024
974867f
correction direct loc
marinebcht Oct 17, 2024
401fd1f
merge
marinebcht Oct 17, 2024
85696b9
inverse ok + compress pas ok
marinebcht Oct 17, 2024
082401b
[DEV][INV] changing the VRT relative path information + small reworks…
Oct 24, 2024
490a858
merge
marinebcht Nov 4, 2024
1026a41
change name SnapDemFileManager to GenericDemFileManager
marinebcht Nov 4, 2024
fa252f2
delecte compression options
marinebcht Nov 8, 2024
03f1aa5
wip test
marinebcht Nov 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
FROM alpine:latest

RUN apk add --no-cache \
openjdk8 \
cmake \
make \
g++ \
gcc \
linux-headers \
proj-dev \
gdal-dev \
git \
curl \
swig \
unzip \
bash \
maven

ENV JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk

ENV LD_LIBRARY_PATH=/usr/lib

# Install GDAL
ARG GDAL_VERSION=3.6.2
RUN mkdir -p /opt/gdal \
&& cd /opt/gdal \
&& curl -LOs https://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz \
&& tar xzf gdal-${GDAL_VERSION}.tar.gz \
&& mkdir build \
&& cd build \
&& cmake \
-DBUILD_JAVA_BINDINGS=ON \
-DJAVA_AWT_INCLUDE_PATH=$JAVA_HOME/include \
-DJAVA_AWT_LIBRARY=$JAVA_HOME/jre/lib/amd64 \
-DJAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/amd64/server \
-DJAVA_INCLUDE_PATH=$JAVA_HOME/include \
-DJAVA_INCLUDE_PATH2=$JAVA_HOME/include/linux \
-DJAVA_AWT_LIBRARY=$JAVA_HOME/lib/amd64 \
-DJAVA_HOME=$JAVA_HOME \
-DCMAKE_INSTALL_PREFIX:PATH=${INSTALL_PREFIX} \
-DCMAKE_PREFIX_PATH:PATH=${INSTALL_PREFIX} \
-DBUILD_SHARED_LIBS:BOOL=ON \
-DCMAKE_BUILD_TYPE=Release \
-DPython_ROOT_DIR=/usr/local \
../gdal-${GDAL_VERSION} \
&& make -j$(nproc) \
&& make install

COPY . /Sen2vm/sen2vm-core

# Copy all needed JAR :
# - orekit v11.3.2
# - rugged v4.0.1
# - SXGEO v0.1.13
# - gdal 3.6.2
RUN mkdir -p /root/.m2/repository/org/ \
&& cp -R /Sen2vm/sen2vm-core/jar/* /root/.m2/repository/org/

# Make a symlink in /usr/lib so we don't need LD_LIBRARY_PATH to load it dynamically from Java.
RUN ln -sf /root/.m2/repository/org/gdal/gdal/3.6.2/libgdalalljni.so /usr/lib

WORKDIR /Sen2vm/sen2vm-core

CMD ["sh"]
Empty file added dock
Empty file.
8 changes: 7 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,11 @@
<type>jar</type>
<optional>false</optional>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.hipparchus</groupId>
<artifactId>hipparchus-core</artifactId>
Expand Down Expand Up @@ -128,6 +133,7 @@
<source>${basedir}/src/main/resources/xsd/GIPP/GS2_GIPP/gs2_spacecraft_model_parameters.xsd</source>
<source>${basedir}/src/main/resources/xsd/GIPP/GS2_GIPP/gs2_viewing_directions.xsd</source>
<source>${basedir}/src/main/resources/xsd/Datastrip/S2_PDI_Level-1B_Datastrip_Metadata.xsd</source>
<source>${basedir}/src/main/resources/xsd/Datastrip/S2_PDI_Level-1B_Granule_Metadata.xsd</source>
</sources>
<xjbSources>
<xjbSource>${basedir}/src/main/resources/binding/jaxb-bindings.xjb</xjbSource>
Expand Down Expand Up @@ -218,4 +224,4 @@
</plugins>
</pluginManagement>
</build>
</project>
</project>
179 changes: 163 additions & 16 deletions src/main/java/esa/sen2vm/Sen2VM.java
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
package esa.sen2vm;

import org.apache.commons.cli.*;
import org.hipparchus.util.FastMath;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.lang.Float;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Vector;
import java.util.HashMap;
import java.util.List;
import java.util.logging.FileHandler;
Expand All @@ -28,17 +32,25 @@
import org.sxgeo.input.dem.SrtmFileManager;
import org.sxgeo.input.dem.GeoidManager;
import org.sxgeo.rugged.RuggedManager;
import org.sxgeo.exception.SXGeoException;

import esa.sen2vm.exception.Sen2VMException;
import esa.sen2vm.input.ConfigurationFile;
import esa.sen2vm.input.GenericDemFileManager;
import esa.sen2vm.input.ParamFile;
import esa.sen2vm.input.datastrip.DataStripManager;
import esa.sen2vm.input.datastrip.Datastrip;
import esa.sen2vm.input.granule.GranuleManager;
import esa.sen2vm.input.granule.Granule;
import esa.sen2vm.input.gipp.GIPPManager;
import esa.sen2vm.input.SafeManager;
import esa.sen2vm.output.OutputFileManager;
import esa.sen2vm.utils.BandInfo;
import esa.sen2vm.utils.DetectorInfo;
import esa.sen2vm.utils.grids.DirectLocGrid;
import esa.sen2vm.utils.grids.InverseLocGrid;
import esa.sen2vm.utils.Sen2VMConstants;

import org.sxgeo.exception.SXGeoException;

/**
* Main class
Expand All @@ -63,7 +75,6 @@ public static final void showPoints(double[][] pixels, double[][] grounds) {
public static void main( String[] args ) throws Sen2VMException
{
Options options = new Options();

Option configOption = new Option("c", "config", true, "Mandatory. Path to the configuration file (in JSON format) regrouping all inputs");
configOption.setRequired(true);
options.addOption(configOption);
Expand All @@ -76,6 +87,8 @@ public static void main( String[] args ) throws Sen2VMException
HelpFormatter formatter = new HelpFormatter();
CommandLine cmd;



try {
cmd = parser.parse(options, args);
} catch (ParseException e) {
Expand Down Expand Up @@ -124,8 +137,8 @@ public static void main( String[] args ) throws Sen2VMException
bands = paramsFile.getBandsList();
}
}
LOGGER.info("detectors = "+detectors);
LOGGER.info("bands = "+bands);
LOGGER.info("Detectors list: " + detectors);
LOGGER.info("Bands list: " + bands);

// Read datastrip
DataStripManager dataStripManager = new DataStripManager(configFile.getDatastripFilePath(), configFile.getIers(), configFile.getBooleanRefining());
Expand All @@ -137,10 +150,9 @@ public static void main( String[] args ) throws Sen2VMException

// Init demManager
Boolean isOverlappingTiles = true; // geoid is a single file (not tiles) so set overlap to True by default
SrtmFileManager demFileManager = new SrtmFileManager(configFile.getDem());
if(!demFileManager.findRasterFile()) {
throw new Sen2VMException("Error when checking for DEM file");
}
GenericDemFileManager demFileManager = new GenericDemFileManager(configFile.getDem());
demFileManager.buildMap(configFile.getDem());

GeoidManager geoidManager = new GeoidManager(configFile.getGeoid(), isOverlappingTiles);
DemManager demManager = new DemManager(
demFileManager,
Expand All @@ -150,7 +162,7 @@ public static void main( String[] args ) throws Sen2VMException
// Build sensor list

// Save sensors for each focal plane
List<Sensor> sensorList = new ArrayList<Sensor>();
HashMap<String, Sensor> sensorList = new HashMap<String, Sensor>();
for (DetectorInfo detectorInfo: detectors) {
for (BandInfo bandInfo: bands) {
SensorViewingDirection viewing = gippManager.getSensorViewingDirections(bandInfo, detectorInfo);
Expand All @@ -169,7 +181,7 @@ public static void main( String[] args ) throws Sen2VMException
msiToFocalplane,
pilotingToMsi
);
sensorList.add(sensor);
sensorList.put(sensor.getName(), sensor);
}
}

Expand All @@ -179,7 +191,7 @@ public static void main( String[] args ) throws Sen2VMException
dataStripManager.getDataSensingInfos(),
Sen2VMConstants.MINMAX_LINES_INTERVAL_QUARTER,
Sen2VMConstants.RESOLUTION_10M_DOUBLE,
sensorList,
new ArrayList(sensorList.values()),
Sen2VMConstants.MARGIN,
dataStripManager.getRefiningInfo()
);
Expand All @@ -193,16 +205,151 @@ public static void main( String[] args ) throws Sen2VMException
demManager
);

double[][] pixels = {{0., 0.}};
double[][] grounds = simpleLocEngine.computeDirectLoc(sensorList.get(0), pixels);
showPoints(pixels, grounds);

LOGGER.info("End Sen2VM");

double[][] pixels_ = {{0., 0.}};
double[][] grounds_ = simpleLocEngine.computeDirectLoc(sensorList.get("B01/D01"), pixels_);
LOGGER.info("pixels = "+pixels_[0][0]+" "+pixels_[0][1]+" grounds = "+grounds_[0][0]+" "+grounds_[0][1]+" "+grounds_[0][2]);

// Safe Manager
SafeManager sm = new SafeManager(configFile.getL1bProduct(), dataStripManager);
Datastrip ds = sm.getDatastrip() ;
//ds.checkNoVRT(detectors, bands) ;

// GIPP
Float georefConventionOffsetPixel = 0.5f;
Float georefConventionOffsetLine = 0.5f;

OutputFileManager outputFileManager = new OutputFileManager();

String epsg = configFile.getReferential();
if (configFile.getOperation().equals(Sen2VMConstants.INVERSE)) {
LOGGER.info("EGSG read from configuration file (for inverse location): " + epsg);
}



LOGGER.info("");
LOGGER.info("Starting grids generation");
for (BandInfo bandInfo: bands) {
LOGGER.info("");
LOGGER.info("");
LOGGER.info("###############");
LOGGER.info("### BAND " + bandInfo.getName() + " ###");
LOGGER.info("###############");
float res = (float) bandInfo.getPixelHeight();
float step = configFile.getStepFromBandInfo(bandInfo);

LOGGER.info("Grid resolution: " + String.valueOf(configFile.getStepFromBandInfo(bandInfo)));
LOGGER.info("Band resolution: " + String.valueOf(res));
System.out.println("det,band,lon,lat,alt");




for (DetectorInfo detectorInfo: detectors) {
//LOGGER.info("");
//LOGGER.info("### DET " + detectorInfo.getName() + " (BAND " + bandInfo.getName() + ") ###");

if (configFile.getOperation().equals(Sen2VMConstants.DIRECT)) {

int[] BBox = sm.getFullSize(dataStripManager, bandInfo, detectorInfo);
int startLine = BBox[0];
int startPixel = BBox[1];
int sizeLine = BBox[2];
int sizePixel = BBox[3];

// Load Granule Info
ArrayList<Granule> granulesToCompute = sm.getGranulesToCompute(detectorInfo, bandInfo);
// LOGGER.info("Number of granules found: " + String.valueOf(granulesToCompute.size()));

// Get Full Sensor Grid
DirectLocGrid dirGrid = new DirectLocGrid(georefConventionOffsetLine, georefConventionOffsetPixel,
step, startPixel, startLine, sizeLine, sizePixel);

double[][] sensorGridForDictorLoc = dirGrid.get2Dgrid(step/2, step/2);

// Direct Loc
double[][] directLocGrid = simpleLocEngine.computeDirectLoc(sensorList.get(bandInfo.getNameWithB() + "/" + detectorInfo.getNameWithD()), sensorGridForDictorLoc);

double[][] pixels_test = {{0., 0.}};
double[][] grounds_test = simpleLocEngine.computeDirectLoc(sensorList.get(bandInfo.getNameWithB() + "/" + detectorInfo.getNameWithD()), pixels_test);
LOGGER.info("DET"+ detectorInfo.getName() + ") pixels = "+pixels_test[0][0]+" "+pixels_test[0][1]+" grounds = "+grounds_test[0][0]+" "+grounds_test[0][1]+" "+grounds_test[0][2]);
System.out.print(detectorInfo.getName() + ",");
System.out.print(bandInfo.getName() + ",");
System.out.print(grounds_test[0][0]);
System.out.print(",");
System.out.print(grounds_test[0][1]);
System.out.print(",");
System.out.println(grounds_test[0][2]);

/*Vector<String> inputTIFs = new Vector<String>();
float pixelOffset = dirGrid.getPixelOffsetGranule().floatValue();
for(int g = 0 ; g < granulesToCompute.size(); g++ ) {
Granule gr = granulesToCompute.get(g) ;
int startGranule = gr.getFirstLine(res);
int sizeGranule = gr.getSizeLines(res);

double[][][] subDirectLocGrid = dirGrid.extractPointsDirectLoc(directLocGrid, startGranule, sizeGranule, configFile.getExportAlt()) ;
float subLineOffset = dirGrid.getLineOffsetGranule(startGranule).floatValue();

// Save in TIF
String gridFileName = gr.getCorrespondingGeoFileName(bandInfo);

// Save with originY = - originY and stepY = -stepY for VRT construction
System.out.println("//");
System.out.println(pixelOffset + step / 2);
System.out.println(-(startGranule + subLineOffset + step / 2));
System.out.println("//");

outputFileManager.createGeoTiff(gridFileName, pixelOffset + step / 2, -(startGranule + subLineOffset + step / 2) ,
step, -step, subDirectLocGrid, "", "EPSG:4326", subLineOffset, pixelOffset) ;

// Add TIF to the future VRT
inputTIFs.add(gridFileName) ;
}

// Create VRT
float lineOffset = dirGrid.getLineOffsetGranule(0).floatValue();
String vrtFileName = ds.getCorrespondingVRTFileName(detectorInfo, bandInfo);
outputFileManager.createVRT(vrtFileName, inputTIFs, step, lineOffset, pixelOffset, configFile.getExportAlt()) ;

// Correction post build VRT
outputFileManager.correctGeoGrid(inputTIFs);
outputFileManager.correctVRT(vrtFileName);*/


} else if (configFile.getOperation().equals(Sen2VMConstants.INVERSE)) {

Float[] bb = configFile.getInverseLocBound() ;
String invOutputDir = configFile.getInverseLocOutputFolder() ;
String nameSensor = bandInfo.getNameWithB() + "/" + detectorInfo.getNameWithD();

// Start
step = step * 100 ; // TODO

InverseLocGrid invGrid = new InverseLocGrid(bb[0], bb[1], bb[2], bb[3], epsg, step);
double[][] groundGrid = invGrid.get2DgridLatLon();

double[][] inverseLocGrid = simpleLocEngine.computeInverseLoc(sensorList.get(bandInfo.getNameWithB() + "/" + detectorInfo.getNameWithD()), groundGrid, "EPSG:4326");
double[][][] grid3D = invGrid.get3Dgrid(inverseLocGrid) ;

String invFileName = ds.getCorrespondingInverseLocGrid(detectorInfo, bandInfo, configFile.getInverseLocOutputFolder());
outputFileManager.createGeoTiff(invFileName, bb[0], bb[1], invGrid.getStepX(), invGrid.getStepY(), grid3D, epsg, "", 0.0f, 0.0f) ;

} else {
LOGGER.info("Operation " + configFile.getOperation() + " does not exist.");
}
}
}


} catch ( IOException exception ) {
throw new Sen2VMException(exception);
} catch ( SXGeoException exception ) {
throw new Sen2VMException(exception);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Loading