Skip to end of metadata
Go to start of metadata

Computer used: HP Z620 workstation

Processor: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz

Ram: 32Gb

Three hdf5 generation programs were created to test chunking and one program for reading them.

 1) hdf5_append.cpp : Creates an hdf5 file by user define images size. It then appends image by image.

 2) hdf5_no_append.cpp : Similar to hdf5_append but instead of appending images, it specifies the whole data set space and performs hyper slab selections to write data to.

 3) hdf5_buffer.cpp : Buffers a number of images in memory before writing the data to the file. The number of images buffered is defined by the chunking in the Z.

 4) hdf5_read.cpp : Reads images as X by Z along the Y axis.

The image size tested was 2048 x 2048 with 1000 images. The data saved is 32 bit integers.

  Dims:

    0 = X

    1 = Y

    2 = Z

Times with Linux time command before running the application. (ex.  bash> time ./hdf5_append.exe 2048 2048 1000 10 10 10 )

HDF5 32 bit Integer Append Writing Table

Chunk X (Height)

Chunk Y (Width)

Chunk Z (Num Images)

Write Run Time (HH:MM:SS)

Memory Used (Mb)

HDF5 File Size (Gb)

Read Y (HH:MM:SS)

1

1

1

N/A (ran for more than 5hours)

23000

N/A

N/A

10

10

1

00:24:46

500

20


10

10

10

00:16:49

455

17

00:05:59

20

20

20

00:06:22

85

16

00:01:40

30

30

30

00:06:29

85

17

00:00:51

40

40

40

00:07:32

60

17

00:00:32

50

50

50

00:08:47

37

16

00:00:24

60

60

60

00:10:27

30

17

00:00:19

70

70

70

01:18:59

30

18

00:00:27

80

80

80

01:21:48

25

17

00:00:23

90

90

90

01:25:58

25

18

00:00:20

100

100

100

01:29:35

42

17

00:00:17

1000

1000

100

01:55:45

20

34

00:01:07

1000

1000

1000

N/A ( stopped at 05:07:43)

20

N/A (30 when stopped)

N/A

2048

2048

1

00:00:44

24

16

00:03:53

2048

2048

10

01:20:45

20

16

 

2048

2048

100

 

 

 

 

2048

2048

200

 

 

 

 

10

2048

10

00:02:20

20

16

 

100

2048

10

01:19:11

20

17

 

100

2048

100

01:23:07

20

17

 

1000

2048

100

01:23:27

20

23

 

HDF5 No Append Writing Table
HDF5 32 bit IntegerBuffer Writing Table

Chunk X (Height)

Chunk Y (Width)

Chunk Z (Num Images)

Run Time (HH:MM:SS)

Memory Used (Mb)

HDF5 File Size (Gb)

Read Y (HH:MM:SS)

1

1

1

N/A

23000

N/A

N/A

10

10

1

00:24:29

500

20

00:05:59

10

10

10

00:03:55

600

17

00:04:23

20

20

20

00:02:13

395

16

00:01:38

30

30

30

00:02:09

521

16

00:00:49

40

40

40

00:01:56

670

17

00:00:32

50

50

50

00:01:47

824

16

00:00:23

60

60

60

00:01:42

975

16

00:00:18

70

70

70

00:01:39

1100

17

00:00:25

80

80

80

00:01:41

1300

16

00:00:20

90

90

90

00:01:37

1400

16

00:00:18

100

100

100

00:01:39

1600

17

00:00:17

1000

1000

100

 

 

 

 

1000

1000

1000

N/A

N/A

N/A

N/A

2048

2048

1

00:00:41

20

16

00:03:55

2048

2048

10

00:00:55

162

16

00:01:30

2048

2048

100

00:01:40

1600

16

00:01:31

2048

2048

200

00:01:40

3100

16

00:02:16

1

2048

100

00:01:45

1600

16

00:01:25

10

2048

100

00:01:40

1600

16

00:02:14

100

2048

100

00:01:40

1600

17

00:01:34

1000

2048

100

00:01:38

1600

23

00:01:32


Changed:

  Data size from 32 bit integer to 16 bit unsigned short

  Grabs average run time for functions

  Dims:

    0 = Z

    1 = Y

    2 = X

Frame size = 8388608 bytes  (2048 x 2048 x 2) (unsigned short = 2 bytes)

1) hdf5_append_16.cpp

2) hdf5_no_append_16.cpp

3) hdf5_buffer_16.cpp

4) hdf5_read_16.cpp

5) test_all.py : script to run hdf5 write tests

Legend:

HH

Hours

MM

Minutes

SS

Seconds

ms

Milliseconds

Gb

Gigabytes

Mb

Megabytes

HDF5 16 bit Unsigned Short Append Writing Table

Chunk X (Height)

Chunk Y (Width)

Chunk Z (Num Images)

Write Run Time (HH:MM:SS)

HDF5 File Size (Gb)

Avg H5Dextend (ms)

Avg H5Dselect (ms)

Avg H5Dwrite / frame (ms)

H5Fclose (ms)

Read Y ?1 (ms)

Read Y NY/2 (ms)

Read Y NY-1 (ms)

Read App Run (ms)

10

10

1

00:09:55

11

256

0.001490

577

59

2709

2567

2330

7610

10

10

10

00:08:21

8.1

256

0.001577

506

60

2721

2276

1674

6718

20

20

20

00:03:19

8.0

256

0.001624

197

3

1549

1507

978

4047

30

30

30

00:02:46

8.2

256

0.001605

161

4

1414

1221

976

3629

40

40

40

00:03:01

8.1

256

0.518378

176

1

1425

1321

1072

3833

50

50

50

00:03:20

7.9

256

0.001624

185

3

1378

1399

1019

3805

60

60

60

00:04:10

8.4

256

0.001597

220

1

1289

1287

1074

3688

70

70

70

00:04:34

8.7

256

0.001623

249

2

1577

1612

1344

4543

80

80

80

00:05:14

8.4

256

0.001666

275

1

1571

1606

1439

4637

90

90

90

00:00:21

8.7

256

0.001518

21

2

2185

2110

1887

6189

100

100

100

00:00:17

8.3

256

0.001609

28

1

2078

1894

2989

7004

200

200

200

00:00:34

9.1

256

0.001511

36

1

3283

2884

3523

9741

300

300

300

00:00:34

9.9

256

0.001404

34

1

4477

4280

5558

14337

1000

1000

10

00:00:46

17

256

0.001559

46

34

16807

16606

16705

50152

1000

1000

100

00:00:25

17

256

0.001621

48

30

16920

16599

16663

50209

1000

1000

1000

00:00:56

17

256

0.001733

55

0

621

1137

1040

2801

2048

2048

1

00:00:11

7.9

256

0.001447

11

0

7645

7414

7244

22326

2048

2048

10

00:00:11

7.9

256

0.001480

10

0

7462

7354

7362

22207

2048

2048

100

00:00:11

7.9

256

0.001490

10

0

7336

7229

7029

21618

2048

2048

200

00:00:11

7.9

256

0.001459

10

0

7519

7379

7380

22291

2048

2048

300

00:00:27

9.4

256

0.001464

27

0

7352

7339

7099

21810

10

2048

10

00:00:52

7.9

256

0.001602

46

732

28268

1922

1444

31653

20

2048

20

00:00:15

8.1

256

0.001444

14

0

27552

26498

134

54218

30

2048

30

00:00:14

8.1

256

0.001448

14

0

28219

27665

1152

57072

40

2048

40

00:00:23

12

256

0.001532

15

0

27386

27147

62

54621

50

2048

50

00:00:29

7.9

256

0.001412

29

0

26722

26439

43

53234

2048

20

20

00:00:31

7.9

256

0.001395

24

1

1294

1103

1003

3496

2048

30

30

00:00:27

8

256

0.001395

24

1

1168

863

1115

3204

2048

40

40

00:00:29

8.1

256

0.001461

29

1

1128

816

1040

3043

2048

50

50

00:00:30

7.9

256

0.001477

30

445

1240

1014

1144

3461

2048

100

100

00:00:28

7.9

256

0.001494

28

1

1506

1447

1786

4797

2048

200

200

00:00:30

7.9

256

0.001448

30

1

2795

2680

2850

8383

2048

300

300

00:00:32

9.4

256

0.001463

32

1

4758

4797

4762

14379

HDF5 No Append 16 bit Unsigned Short Writing Table

Chunk X (Height)

Chunk Y (Width)

Chunk Z (Num Images)

Write Run Time (HH:MM:SS)

HDF5 File Size (Gb)

Avg H5Dselect (ms)

Avg H5Dwrite (ms)

H5Fclose (ms)

Read Y ?1 (ms)

Read Y NY/2 (ms)

Read Y NY-1 (ms)

Read App Run (ms)

10

10

1

00:10:13

11

0.001490

577

62

2709

2567

2330

7610

10

10

10

00:08:21

8.1

0.003219

495

59

2702

2374

1728

6824

20

20

20

00:03:12

8

0.003190

189

1

1641

1525

1046

4229

30

30

30

00:02:47

8.2

0.002999

163

477

1547

1258

992

3812

40

40

40

00:03:03

8.1

0.002930

181

1

1483

1370

1129

3996

50

50

50

00:03:20

7.9

0.002832

189

2

1523

1412

1128

4071

60

60

60

00:04:05

8.4

0.002864

229

1

1380

1431

1286

4114

70

70

70

00:04:47

8.7

0.002822

259

547

1576

1686

1451

4641

80

80

80

00:05:10

8.4

0.002885

284

1

1818

1717

1576

5126

90

90

90

00:00:22

8.7

0.002882

20

1

1935

1995

2018

5959

100

100

100

00:00:27

8.3

0.003056

24

1

2124

1657

2948

6761

200

200

200

00:00:39

9.1

0.003082

31

6

3613

3128

3849

10649

300

300

300

00:00:37

9.9

0.003222

31

1

5049

4784

6206

16061

1000

1000

10

00:00:52

17

0.003522

52

35

18403

18723

18580

55663

1000

1000

100

00:00:54

17

0.003227

43

34

18140

18539

18617

55323

1000

1000

1000

00:00:55

17

0.003394

55

247

713

1301

1244

3262

2048

2048

1

00:00:27

7.9

0.003167

27

0

3447

3349

3215

10036

2048

2048

10

00:00:27

7.9

0.003263

26

31

3954

4023

3975

11992

2048

2048

100

00:00:26

7.9

0.003316

26

0

4051

4264

4022

12353

2048

2048

200

00:00:25

7.9

0.002937

16

0

4085

4165

4059

12324

2048

2048

300

00:00:26

9.4

0.003131

25

0

4199

4315

4190

12727

10

2048

10

00:00:51

7.9

0.002727

49

2707

7187

1469

1378

10037

20

2048

20

00:00:19

7.9

0.002867

18

317

1092

29608

136

30840

30

2048

30

00:00:22

8.1

0.002957

22

0

1416

30815

1197

33461

40

2048

40

00:00:25

8

0.003099

25

0

1582

29734

60

31394

50

2048

50

00:00:25

7.9

0.003175

25

0

2331

28716

49

31108

2048

20

20

00:00:35

7.9

0.003297

33

2487

1572

1034

1001

3610

2048

30

30

00:00:34

8.1

0.003046

28

2344

1048

930

1024

3016

2048

40

40

00:00:30

8

0.003170

30

1

1020

864

1123

3058

2048

50

50

00:00:34

7.9

0.003283

34

0

1326

1006

1353

3741

2048

100

100

00:00:25

8.1

0.003498

25

1

1705

1613

2016

5343

2048

200

200

00:00:26

8.4

0.003197

26

2

3215

3121

3253

9647

2048

300

300

00:00:26

9.7

0.003207

26

0

4774

5492

5242

15522

HDF5 16 bit Unsigned Short Buffer Writing Table

Chunk X (Height)

Chunk Y (Width)

Chunk Z (Num Images)

Write Run Time (HH:MM:SS)

HDF5 File Size (Gb)

Avg H5Dselect (ms)

Avg H5Dwrite (ms)

Time to Write 1 frame (ms)

Write buffer size (Mb)

H5Fclose (ms)

Read Y ?1 (ms)

Read Y NY/2 (ms)

Read Y NY-1 (ms)

Read App Run (ms)

10

10

1

00:09:05

11

0.003630

529

529.831465

8

62

2649

2692

2345

7690

10

10

10

00:01:16

8.1

0.000375

70

7.047783

80

2

2144

2007

1659

5814

20

20

20

00:00:25

8.0

0.000191

25

1.261457

160

13

1641

1736

2179

5559

30

30

30

00:00:21

8

0.000122

20

0.678740

240

2

1515

1490

2562

5570

40

40

40

00:00:22

7.9

0.000092

16

0.413571

320

4

1762

1570

1979

5315

50

50

50

00:00:24

7.9

0.000090

9

0.199430

400

5

1462

1405

1550

4419

60

60

60

00:00:27

7.9

0.000066

11

0.192680

480

616

1671

1589

1266

4540

70

70

70

00:00:27

8.1

0.000069

6

0.089154

560

2

1590

1610

1401

4613

80

80

80

00:00:25

7.8

0.000061

3

0.046679

640

17

1823

1711

1637

5179

90

90

90

00:00:26

8

0.000059

5

0.057209

720

374

1709

1703

1682

5102

100

100

100

00:00:27

8.3

0.000051

5

0.057432

800

1

2104

1752

2600

6459

200

200

200

00:00:34

9.1

0.000038

3

0.015676

1600

3

3684

3669

4451

11814

300

300

300

00:00:27

7.4

0.000032

2

0.006680

2400

0

4824

4661

6213

15715

1000

1000

10

00:01:05

17

0.000343

60

6.096869

80

40

20910

21085

20910

62931

1000

1000

100

00:00:45

17

0.000049

4

0.044514

800

25

21180

21198

21644

64040

2048

2048

1

00:00:28

7.9

0.003258

28

28.741258

8

0

6294

6219

6125

18652

2048

2048

10

00:00:28

7.9

0.000359

27

2.794249

80

0

5888

5826

5768

17495

2048

2048

100

00:00:27

8.1

0.000053

6

0.061163

800

0

5936

5842

5851

17640

2048

2048

200

00:00:26

8.4

0.000040

1

0.007984

1600

0

5934

5877

5961

17786

2048

2048

300

00:00:23

7.1

0.000032

1

0.004560

2400

0

5183

5368

5159

15721

10

2048

10

00:00:33

7.9

0.000314

23

2.368244

80

809

40987

4505

1390

46897

20

2048

20

00:00:21

7.9

0.000180

21

1.079292

160

0

39555

38553

129

78264

30

2048

30

00:00:37

7.9

0.000129

15

0.503803

240

5348

39040

38701

1113

78898

40

2048

40

00:00:46

8.0

0.000099

16

0.422420

320

9783

39381

39084

58

78541

50

2048

50

00:00:37

7.9

0.0000 87

9

0.188128

400

6267

38786

38598

50

77461

2048

20

20

00:00:37

7.9

0.000181

30

1.528609

160

2910

1427

1340

1070

3905

2048

30

30

00:00:38

7.9

0.000128

13

0.435905

240

326

1355

1104

1172

3652

2048

40

40

00:00:42

8.0

0.000102

14

0.364752

320

0

1519

1066

1257

3872

2048

50

50

00:00:43

7.9

0.000085

6

0.132730

400

0

1554

1314

1463

4397

2048

100

100

00:00:42

8.1

0.000066

4

0.044710

800

1

2070

2042

2437

6614

2048

200

200

00:00:43

8.4

0.000033

2

0.010311

1600

2

3843

3891

3927

11737

2048

300

300

00:00:41

7.3

0.000030

1

0.004134

2400

0

5846

5604

5622

17094

  • No labels